My code:

from flytekit import task, workflow, ImageSpec

image_spec = ImageSpec(packages=["numpy", "tensorflow"], base_image="<http://container-image-registry.corp.linkedin.com:8083/lps-image/linkedin/mldev-workspace/mldev-base-image:0.0.16|container-image-registry.corp.linkedin.com:8083/lps-image/linkedin/mldev-workspace/mldev-base-image:0.0.16>", pip_index="<https://lerna.tools.corp.linkedin.com/pypi/simple/>", registry="localhost:30000")

@task(container_image=image_spec)
def t1() -> str:
    import numpy as np
    print("task 1")
    print(np.array([1,2,3]))

    import tensorflow as tf
    print("task 2")
    print(tf.constant(8))
    return("blah1")

@workflow
def wf():
    t1()

Then I run:

pyflyte run --remote example.py wf

Error:

Failed with Unknown Exception <class 'Exception'> Reason: failed to build the imageSpec at /var/folders/9v/3dl8rh2x0nx9ccbv3xkd708h002_42/T/flyte-mktmp10q/sandbox/local_flytekit/4e7f726c4f5a035e23addb662302e8e1/build.envd with error b'error: failed to do request: Head "<http://localhost:30000/v2/flytekit/blobs/sha256:a0e77e7122fc5edfccf28327cc28f2c15e34ee2ec76fa0c7486a6679c014a134>": dial tcp 127.0.0.1:30000: connect: connection refused\n'

[1/1] currentAttempt done. Last Error: USER::containers with unready status: [primary]|context deadline exceeded

I’ve seen a few times such error. What would be the root cause of it?

Hi, I am very new to the Flyte and trying to learn how flyte compiles its workflow and there’s a cli: pyflyte serialize, but I would not get it work with an example project, created via pyflyte init. It will be great if anyone can share some insights. Here is my question in the SO: https://stackoverflow.com/questions/76298549/how-to-run-pyflyte-serialize-workflows

Hi Team, I have noticed when users are logged in to Flyte Console via Auth0, any workflows launched from Console go to an

UNKNOWN

state. I was wondering what could be possible cause for this? Do we need to inform any service other than Flyte Admin about the Auth? For Example Flyte Propeller? CC: @Prafulla Mahindrakar

Hello folks! Suppose we have Flyte workflow with some failed map_task. At the start, this task submits a lot(300-500) pods to Kubernetes, and one of them dies, so the whole workflow is failed. But the rest pods are still in Kubernetes and waiting to be executed. Why does Flyte not kill the rest pods if one of them is failed? To be able to rerun this workflow, all these pods should be manually deleted from the Kubernetes cluster.

Hi Community, I tried to integrate self-hosted Gitlab as an external OAuth2 Authorization Server but stuck at

CSRF_TOKEN_VALIDATION_FAILED

, any clues on how to debug the problem? The flyte is deployed on self-hosted k8s cluster with an internal domain name and HTTP access (not HTTPS) using template generated by

flyte-core

helm chart (

helm template flyte-core -f values.yaml

). The OIDC seems to work (After a fresh deployment, when opening flyte console, it will redirect to the gitlab authorization page, and after approval it can redirect back to flyte console page with username show on the top-right corner), but when I try to register workflow according to the tutorial [1] with command

flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1

, the gitlab authorization page is prompted successfully but when redirect back to

<http://flyte.example.com/callback|flyte.example.com/callback>

page, it complains about

401

error. And the log of

flyteadmin

shows something like

{"json":{},"level":"error","msg":"Invalid CSRF token cookie [CSRF_TOKEN_VALIDATION_FAILED] CSRF token does not match state 2r4rcd3npg, 3237e1083ec0ae2bd20acbe8a5817d18475faaee5a060d2184ab7ffddd151290 vs OXpoczQyanRxcW43c3hnZ3RjbnBnZjZrNnptMnA2dDY","ts":"2023-01-29T06:43:34Z"}

There are several small questions that might be relevant: 1. In the auth doc [2], there is a comment for

redirectUri: <http://localhost:53593/callback>

saying that

This should not change

, but I suppose it is for sandbox deployment so I changed it to the domain name something like

<http://flyte.example.com/callback>

. Otherwise, it will complains about invalid redirect uri when authorize on gitlab page. 2. For the

scopes

, according to the gitlab doc [3], I only set it to

read_user

. 3. Any approach to show more logs for

flytectl

CLI? I tried to set the log level to

0

, but it does not show anything. 4. Any approach to show more logs for

flyteadmin

service? I found there is a

flyteadmin.extraArgs

in the template but does not know how to inject

--logger.level 0

. The relevant

values.yaml

for auth is shown as following (with some sensitive info masked):

configmap:
  adminServer:
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          metadataUrl: .well-known/openid-configuration
        thirdPartyConfig:
          flyteClient:
            clientId: <client_id generated by gitlab>
            redirectUri: <http://flyte.example.com/callback>
            scopes:
              - read_user
      authorizedUris:
        - <http://flyte.example.com>
        - <http://flyteadmin:80>
        - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://git.example.com>
          clientId: <client_id generated by gitlab>
          scopes:
            - openid
    server:
      security:
        useAuth: true
flyteadmin:
  secrets:
    oidc_client_secret: <client_secret generated by gitlab>
secrets:
  adminOauthClientCredentials:
    clientId: <client_id generated by gitlab>
    clientSecret: <client_secret generated by gitlab>

[1] https://docs.flyte.org/projects/cookbook/en/latest/auto/larger_apps/larger_apps_deploy.html#build-deploy-your-application-to-the-cluster [2] https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html [3] https://docs.gitlab.com/ee/integration/oauth_provider.html#view-all-authorized-applications

I'm trying to understand the difference between task and workflow at a fundamental level. My understanding was that tasks could not call other tasks. I wrote the following dummy example:

@task
def test(x: int) -> int:
    return x + 1

@task
def echo_integer(value: int) -> int:
    logging.error(f"I've got value {value}")
    return test(x=value + 5) + 1

@workflow
def wf(
    value: int,
):
    return echo_integer(value=value)

When run locally, we see type-errors because the

echo_integer

task now returns a promise instead of an int. Interestingly, it works fine when we run remotely, with the caveat that the called task is not run as a isolated container. I would have expected it to fail remotely as well. Any pointers about what I'm missing in my understanding?

Hello! I’m trying to convert a Python dict to a LiteralMap and found the DictTransformer class. It states:

Transformer that transforms a univariate dictionary Dict[str, T] to a Literal Map

But it only seems to be able to transform it into a Literal. So how do I go from a Literal to a LiteralMap?

Hi team.. Can

dynamic

be cached?

Hello, I’m trying to run the Getting Started wine classification example on my Flyte setup, but have faced several errors. My setup is consisted of: Jupyter Hub + Flyte on the same K8S cluster. The codes are

from flytekit.remote import FlyteRemote
from flytekit.configuration import Config, PlatformConfig, ImageConfig, SerializationSettings

remote = FlyteRemote(
    config=Config(
        platform=PlatformConfig(
            endpoint="dns:///flyte-backend-flyte-binary-grpc.namespace.svc.cluster.local:8089",
            insecure=True,
            insecure_skip_verify=True,
        )
    ),
    default_project="flytesnacks",
    default_domain="development",
    data_upload_location="<s3a://flyte-storage/>"
)

HYPERPARAMS = {
    "hyperparameters": {"C": 0.1}
}

flyte_entity = training_workflow

flyte_workflow = remote.register_workflow(
    entity=flyte_entity,
    serialization_settings=SerializationSettings(image_config=ImageConfig.auto_default_image()),
    version="v1"
)

launch_plan = LaunchPlan.get_or_create(
    workflow=flyte_workflow,
    name="wine_classification_lp",
    default_inputs=HYPERPARAMS,
)
remote.register_launch_plan(launch_plan, "v1", project="flytesnacks", domain="development")

And the workflow failed right at the first task

get_data

, here is the error I’ve got

[1/1] currentAttempt done. Last Error: USER::rapped(*args, **kwargs)                        │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py:347 in     │
│ _execute_task                                                                │
│                                                                              │
│ ❱ 347 │   │   _task_def = resolver_obj.load_task(loader_args=resolver_args)  │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/flytekit/core/utils.py:295 in wrapper │
│                                                                              │
│ ❱ 295 │   │   │   │   return func(*args, **kwargs)                           │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/flytekit/core/python_auto_container.p │
│ y:235 in load_task                                                           │
│                                                                              │
│ ❱ 235 │   │   task_module = importlib.import_module(name=task_module)  # typ │
│                                                                              │
│ /usr/local/lib/python3.8/importlib/__init__.py:127 in import_module          │
│                                                                              │
│ ❱ 127 │   return _bootstrap._gcd_import(name[level:], package, level)        │
│ in _gcd_import:1011                                                          │
│ in _sanity_check:950                                                         │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: Empty module name

Any suggestion on which might have caused this error and how to fix it? I’m suspecting that I haven’t registered the workflow or the launch plan properly.