Hi Flyte friends, I am using pyflyte and flytectl to package and push workflows to my cluster. I can see the pushed workflows through flyte console When I attempt to launch any workflow, the first task fails after about 40 seconds with the error in the picture.

Hi folks, I am running into an error when running a simple hello_world workflow. I am using a custom docker image. Any idea why?

FROM python:3.8-buster

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

ARG tag
ARG wandb_api_key
ARG wandb_username

ENV FLYTE_INTERNAL_IMAGE $tag
ENV WANDB_API_KEY $wandb_api_key
ENV WANDB_USERNAME $wandb_username

# Install the AWS cli separately to prevent issues with boto being written over
RUN pip3 install awscli

RUN apt-get update && apt-get install -y curl

ENV VENV /opt/venv
# Virtual environment
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"

# Install Python dependencies
COPY requirements.txt /root/.
RUN pip install -r /root/requirements.txt

# Copy the actual code
COPY src/ /root/src/

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[ab8dm4nttv49wgvn59kg-n0-0] terminated with exit code (1). Reason [Error]. Message: 
rtlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'site-packages.flytekit'
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 513, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://senn-ai-mlops-flyte/metadata/propeller/flytesnacks-development-ab8dm4nttv49wgvn59kg/n0/data/inputs.pb>', '--output-prefix', '<s3://senn-ai-mlops-flyte/metadata/propeller/flytesnacks-development-ab8dm4nttv49wgvn59kg/n0/data/0>', '--raw-output-data-prefix', '<s3://senn-ai-mlops-flyte/data/sk/ab8dm4nttv49wgvn59kg-n0-0>', '--checkpoint-path', '<s3://senn-ai-mlops-flyte/data/sk/ab8dm4nttv49wgvn59kg-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://senn-ai-mlops-flyte/mo/flytesnacks/development/NRI2T5OSZMCFXKR4CUNLWO7MSM======/fastab502ef8d4ae75b6b5497a94633e8642.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'site-packages.flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'src.workflows.hello_world', 'task-name', 'say_hello']' returned non-zero exit status 1.

Is this a known issue? E: Fixed

$ docker pull <http://cr.flyte.org/flyteorg/flytescheduler-release:v1.4.3|cr.flyte.org/flyteorg/flytescheduler-release:v1.4.3>
5e5f900059c8: Download complete
0f96f76f982d: Download complete
88dae1745799: Downloading [>                                                  ]       0B/2.343kB
failed to copy: httpReadSeeker: failed open: failed to do request: Get "<https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:88dae1745799f4078f859b8ada55bc1f1757d10d9c97cdeb62f88cfc827c3a1f?se=2023-03-24T21%3A05%3A00Z&sig=rMhv3oYchaYycF260vI0pdx6pENdYsKpph9eXak%2BMPw%3D&sp=r&spr=https&sr=b&sv=2019-12-12>": x509: certificate is valid for *.<http://githubassets.com|githubassets.com>, <http://githubassets.com|githubassets.com>, not <http://pkg-containers.githubusercontent.com|pkg-containers.githubusercontent.com>

Hi, we can specify the number of gpus in flyte tasks, but how do we specify which type of gpu to use?

Hi, I was trying to use a local

csv

file. But I got this error:

Message:

    [Errno 2] No such file or directory: 'data.csv'

User error.

What would be the workaround?

Hi, guys! Could you help with s3 credentials. I can run hello_world example in sandbox env, but I am failing to do so on our local k8s cluster.

raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://blabs-flyte-mgmt-metadata/dispute-resolution/development/BP7IZUPWYESUYFGRO52LE7HBDU======/scriptmode.tar.gz> to /root/ (recursive=False).

Original exception: Access Denied

We don’t have any auth yet for flyte. I assume I just haven’t specified s3 bucket endpoint, access_key_id, and secret_access_key. Where should I do this? In my

.flyte/config.yaml

? Or on

k8s/values.yaml

side via devops support? If so what is the template?

storage:
  access-key: <SECRET>
  auth-type: iam
  disable-ssl: false
  endpoint: <s3://blabs-flyte-mgmt-metadata>
  region: eu-central-1
  secret-key: <SECRET>

I don’t understand how to use this page https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.configuration.S3Config.html

Hi! I'm evaluating Flyte for our workflow orchestration needs. I'm curious what will happen when WorkflowExecutor tries to update the task status to FlyteAdmin but FlyteAdmin is unavailable for a prolonged duration (say 1 hour) for some reason. Will the WorkflowExecutor eventually update the correct task status to the FlyteAdmin when it comes back up? Or will the FlyteAdmin have an inconsistent state (say Running when the task has actually failed/succeeded when the FlyteAdmin was down)

what constitutes as a valid execution name? I am having a hard time finding the validator both in the docs and code

Error: rpc error: code = InvalidArgument desc = invalid name format: 370eb303160612

x-posting from #flyte-deployment: https://flyte-org.slack.com/archives/C01P3B761A6/p1679938476764839

Hi team, What is the suggestions/do’s & don’t for sharing flyte tasks across multiple workflows? WorkflowA has a taskA, registered with SHA1 WorkflowB uses taskA and has the same SHA1, when we register both workflows, get the error that task_id already exists with SHA1. How can we get around this issue?

Hello, Do you know if there is a flytekit plugins for airflow? I need to detect if an Airflow dag run is completed from Flyte. Thanks.

Hi all, I have a task returning a

StructuredDataset

that's failing if I turn on

cache_serialize

with this error:

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/opt/venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 572, in dispatch_execute
        raise TypeError(
    Failed to convert return value for var o0 for function {my task} with error <class 'pyarrow.lib.ArrowInvalid'>: ("Could not convert ('... ... (580 characters truncated) ... ...',) with type Row: did not recognize Python value type when inferring an Arrow data type", 'Conversion failed for column sequence with type object')

The sequence column of the dataframe I return is a string (no non-standard characters). Any idea on what could be causing this? Regular (no

cache_serialize

) cache works just fine.

@Prashant Jain has left the channel

flyteidl 1.3.12 depends on protobuf<5.0.0 and >=4.21.1
    sagemaker 2.133.0 depends on protobuf<4.0 and >=3.1

Since 1.3 FlyteKit has version conflict with Sagemaker API and Tensorflow as well by deprecating protobuf 3 making it impossible to use those libraries together

Hey, does anybody know how can i pass multiple values to task_config param in @task decorator i.e. i want to pass Pod configuration as well as Spark configuration?

Hi everyone! I'm trying to run the workflow remotely as described in the Flyte Fundamentals docs, but I get an OpenSSL version error:

E0328 20:15:47.445709000 140704578418304 <http://ssl_transport_security.cc:1495]|ssl_transport_security.cc:1495]> Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

I'm using Python 3.9, grpcio & grpc-status 1.48.2, pyOpenSSL 23.0.0 and against a local flyte-sandbox (latest version) Can anyone help? I'm trying to get this working as a proof-of-concept for wider use in my compnay and need to demonstrate that this works

Hello - I've been looking through the docs and searching here, and haven't figured out how, from flytekit, to query a flyte set up to get the execution status of a workflow that was started via FlyteRemote.emote.execute. Essentially, I want to be able to display in the system that kicked off the workflow if it's Pending, Running, Succeeded, etc. It seems like it should be possible. If someone could point me in the right direction, I'd be most appreciative.

Hey all! With GCP, the logs before user code is executed are all coming up as severity=ERROR. I'd love to try to fix this at some point and was wondering if anyone has thoughts! Ideally this could be configurable too - I was talking to @Yee about this earlier and he suggested posting here for visibility! I also made a GitHub issue

Hey I've installed flyte on GKE it was running fine till now but i wanted to increase ResourceQuota for development/ staging and production namespace so i changed the config for the same but suddenly flyte-admin pod sync-cluster-resource container is failing with this error.

panic: Failed to sync cluster resources : no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>"

goroutine 1 [running]:
main.main()
/go/src/github.com/flyteorg/flyteadmin/cmd/main.go:13 +0x91

Hello all I'm having an issue when trying to use reference tasks... When building image, I'm getting this error:

'ReferenceTask' object has no attribute '_task_function'

... Can someone help me? I would be very appreciated

Hello all, I could find here that Authorization can be set up for Flyte. But in the doc in production grade Flyte deployment, its mentioned that "Authorization is not supported out-of-the-box in Flyte". Is this just for UI or am I missing something here. So does Flyte have Role based access control and Authorization support or can it be achieved, thinking in terms of production requirements?

hey! I'm back with another question 😅 Can multiple instances of FlyteAdmin be run simultaneously without issues (such as duplicated scheduled runs)? I found the following about the data plane but nothing about FlyteAdmin scalability_._ While FlyteAdmin serves the Workflow Execution API, it does not itself execute workflows. To launch workflow executions, FlyteAdmin sends the workflow DAG to the DataPlane. For added scalability and fault-tolerance, FlyteAdmin can be configured to load-balance workflows across multiple isolated data-plane clusters.

@David Espejo (he/him) showed a really insightful diagram in one of the recent meetings. Recording on Youtube at . Is that diagram available someplace? The data plane is not visible in the recording and that architecture diagram is super useful to new users.

is it possible to run a read only FlyteAdmin replica (i.e drop all write requests, do not schedule runs) without patching the server?

Hi team, If we add labels/annotations to a pod in K8, can we also archive those as part of Flyte metadata tables?

Hi ! I am trying to override my preset k8s specs of a task inside a dynamic workflow and cannot find a way to make it correctly. I figured that it could come from the container name not matching the one that is being created for the given task. Do you know how to do it ? Here is the code that I used :

if training_args.force_a100_gpus:
        return train(input_args).with_overrides(
            pod_template=PodTemplate(
                pod_spec=V1PodSpec(affinity=V1Affinity(
                    node_affinity=V1NodeAffinity(
                        required_during_scheduling_ignored_during_execution=V1NodeSelector(
                            node_selector_terms=[
                                V1NodeSelectorTerm(match_expressions=[
                                    V1NodeSelectorRequirement(
                                        key="<http://cloud.google.com/gke-accelerator|cloud.google.com/gke-accelerator>", operator="In",
                                        values=["nvidia-tesla-a100"])])]
                        ))),
                    restart_policy='never',
                    containers=[V1Container(name='primary',
                                            image='{{.image.imagename.fqn}}:{{.image.imagename.version}}',
                                            resources=V1ResourceRequirements(limits={"<http://nvidia.com/gpu|nvidia.com/gpu>": '1'},
                                                                             requests={"memory": "...",
                                                                                       "cpu": "..."})
                                            )])
            ),
            container_image=None,
            requests=None,
        )

Container name for the train task was "<executionID>-<workflowID>-0-dn2-0"

Does anyone know the right way to raise a

ContainerError

inside a task using flytekit python? https://docs.flyte.org/projects/flyteidl/en/latest/protos/docs/core/core.html#ref-flyteidl-core-containererror For example, if I want to raise/return

NON_RECOVERABLE

or

RECOVERABLE

depends of the task logic

Hi everyone! Comparisons are always hard and tend to be unfair (unless there are clear goals). If anyone here could chime in on this conversation to help guide a user comparing two solutions, that'd be great.

@Yee let's continue the conversation here

Hi team, I get the following error when I try to pass an array (which is a List[DataClass]) of length 15000 to a task.

TooLarge: Event message exceeds maximum gRPC size limit, caused by [rpc error: code = ResourceExhausted desc = grpc: received message larger than max (6020936 vs. 4194304)].

How can I address this? where can I increase this size limit and how large is the recommended size be?