here are the logs for excited container..

cc: @Prafulla Mahindrakar @Yuvraj, @Yash Panchwatkar isn’t able to start a demo cluster on Ubuntu and RHEL. 1. On Ubuntu, the above screenshot explains the error. 2. On RHEL, the error is

context default doesn't exist

. Reference:

Hi! We are looking into different alternatives how to run DBT with Flyte and came across this plugin: https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dbt Do anyone know the status of it and is it the recommended approach? The alternative we were thinking about is to run a docker image which execute the dbt command. Or maybe there are any other better way of running DBT?

This is more of a Pytest question rather than a Flyte one. I’m looking to see if there’s a preferred way of setting default values in

SecretsManager

within tests with the goal of not accidentally using secrets in

/etc/secrets

. We have a way of doing this now (I’ll add what we have in 🧵) but was hoping to see if there’s a standard way of doing this

What is the advice on using caching on entire workflows? I don't see any docs on that, just for tasks. My workflow doesn't return any values at the end, so I suppose I could prepend a task at the beginning of the workflow whose entire purpose is to handle the cache (i.e. have the same inputs as the workflow and add

cache=True

). Is there a better way? What about for workflows that do return values?

Hello, I have opened many PR's in UnionML repository, please anyone review those PR's. I opened the PR's on Oct 1, till now no one responded.

Hi - I asked this question on a different channel but I believe this is the right channel for this question. Can someone help with this? https://flyte-org.slack.com/archives/C01P3B761A6/p1664854331557839

@Navyansh Kesarwani, could you let us know the commands you ran?

Maybe this is a stupid question. But is it possible to somehow within the workflow create dependencies between tasks without connecting them with input and output? example:

from flytekit import task, workflow


@task
def task_a() -> None:
  print("task_a")


@task
def task_b() -> None:
  print("task_b")


@task
def task_c() -> None:
  print("task_c")


@workflow
def my_wf() -> None:
  task_a()
  task_b()
  task_c()

this workflow would run task_a, task_b and task_c in parallel. But i want them to run sequentially.

Hi, I think the enabled/disabled status of the Google Analytics feature in the console is being displayed incorrectly. I created an issue on the flyteconsole repo. Is there a process to mention it on slack or make it visible in some other way? I'm unable to label it or attach a screenshot.

Hey, can Flyte handle

field

defaults on dataclasses? Dataclasses can't have list / dict / set defaults (https://stackoverflow.com/q/52063759) and you have to resort to something like

str_list: List[str] = field(default_factory=lambda: ["a", "b"])

instead, but on Flyte console when launching a workflow it just shows an empty list for me.

Hi, can we specify workflow version in LaunchPlan.get_or_create method.

cron_lp = LaunchPlan.get_or_create(
    name="cron_scheduler",
    workflow=scheduler_workflow,
    schedule=CronSchedule(
        schedule="*/1 * * * *",
    ),
    default_inputs={'f': 4.5},
)

Are there any major “gotchas” to upgrading from a much older flytepropeller/flyteplugins to a more recent version? Are there any changes required for users who register their workflows (i.e. upgrading flytekit or anything like that)?

Hey 👋🏻 , Has anyone made the flyte deck work for notebook tasks? I got a 404 after a successful run Also, is there a way to copy the executed notebook ipynb or html to GCS ? I got file not found when I tried to use the implicit output of notebook task

No such file or directory: '/v3/f25eaabfb23924c25946-n1-0/d1a84d1c0cea75f237d21a70dcde6810/model1-out.ipynb'

Hi, trying to plan out the developer experience for people working in a flyte setup where we'd be making heavier use of pre-registered workflows/tasks (reference_task decorator). That pushes it towards using local k8s which is fine but my first experiments suggest it's pretty slow, adding something like 10+s per task. Any guide as to why that might be?

Hi everyone! I was wondering where i could look inside k8s to verify that a resource quota had been updated?

Hello, I have a flyte .py file that imports a module I created from a subdir called lib and it contains another workflow that is intended as subflow.

from lib.subwf import my_subwf

I can run the flyte flow successfully in local mode. However, it errored when running with --remote in the demo flyte cluster I started on my PC. The error msg is:

ModuleNotFoundError: No module named 'lib'

I am not able to find flyte documentation for the resolution of this issue. Could anyone share the root cause of the error and the fix?

Thank you!

How can I register my code( written using flytekit ) in flyte sandbox and run it from remote (executing wf from a python code)

I wrote the following TypeTransformer for Light GBM Dataset and am seeing it default to pickle on registering the tasks and workflow that use a Light GBM Dataset. Would any be able to help point me in the right direction to troubleshoot this?

class LGBMDatasetTransformer(extend.TypeTransformer[lgb.Dataset]):
    _TYPE_INFO = flytekit.BlobType(
        format="binary", dimensionality=flytekit.BlobType.BlobDimensionality.MULTIPART
    )

    def __init__(self) -> None:
        super(LGBMDatasetTransformer, self).__init__(name="lgbdataset-transform", t=lgb.Dataset)

    def get_literal_type(self, t: Type[lgb.Dataset]) -> flytekit.LiteralType:
        return flytekit.LiteralType(blob=self._TYPE_INFO)

    def to_literal(
        self,
        ctx: flytekit.FlyteContext,
        python_val: lgb.Dataset,
        python_type: Type[lgb.Dataset],
        expected: flytekit.LiteralType,
    ) -> flytekit.Literal:
        """
        Convert from the given python type object ``lgb.Dataset`` to the Literal representation.
        """
        # Step 1: Upload all the data into a remote place recommended by Flyte

        local_file = ctx.file_access.get_random_local_path() + ".data"
        lgb.Dataset.save_binary(python_val, local_file)

        remote_dir = ctx.file_access.get_random_remote_directory()
        ctx.file_access.upload_directory(local_file, remote_dir)
        # Step 2: Return a pointer to this remote_dir in the form of a Literal
        return flytekit.Literal(
            scalar=flytekit.Scalar(
                blob=flytekit.Blob(
                    uri=remote_dir, metadata=flytekit.BlobMetadata(type=self._TYPE_INFO)
                )
            )
        )

    def to_python_value(
        self,
        ctx: flytekit.FlyteContext,
        lv: flytekit.Literal,
        expected_python_type: Type[lgb.Dataset],
    ) -> lgb.Dataset:
        """
        In this method, we want to be able to re-hydrate the custom object from Flyte Literal value.
        """
        # Step 1: Download remote data locally
        local_dir = ctx.file_access.get_random_local_directory()
        ctx.file_access.download_directory(lv.scalar.blob.uri, local_dir)
        # Step 2: Create the ``lgb.Dataset`` object
        return lgb.Dataset.construct(lgb.Dataset(local_dir))


extend.TypeEngine.register(LGBMDatasetTransformer())

Hi all, the task run log is external link and flyte doesn’t collect and show the log real time on the webui, also flyte doesn’t persist the log. the log is gone after the pod destruction, any idea how to handle this?

hello guys .. i am trying to start sandbox environment in k8s cluster but minio pod is goes in crashloopbackoff again and again could you please help me to find what's getting wrong.. here.

result from describe command for minio

Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  16m                 default-scheduler  Successfully assigned flyte/minio-5745464bb-zthw8 to ip-172-31-31-189.ap-south-1.compute.internal
  Normal   Pulling    16m                 kubelet            Pulling image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>"
  Normal   Pulled     16m                 kubelet            Successfully pulled image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" in 13.816089274s
  Normal   Created    14m (x5 over 16m)   kubelet            Created container minio
  Normal   Started    14m (x5 over 16m)   kubelet            Started container minio
  Normal   Pulled     14m (x4 over 16m)   kubelet            Container image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" already present on machine
  Warning  BackOff    85s (x61 over 15m)  kubelet            Back-off restarting failed container

and result from logs

[root@ip-172-31-31-189 ~]# kubectl logs  minio-5745464bb-zthw8 -n flyte
 07:15:01.89
 07:15:01.90 Welcome to the Bitnami minio container
 07:15:01.90 Subscribe to project updates by watching <https://github.com/bitnami/bitnami-docker-minio>
 07:15:01.90 Submit issues and feature requests at <https://github.com/bitnami/bitnami-docker-minio/issues>
 07:15:01.90
 07:15:01.90 INFO  ==> ** Starting MinIO setup **
 07:15:02.01 INFO  ==> Starting MinIO in background...
 07:15:12.01 INFO  ==> Adding local Minio host to 'mc' configuration...
 07:15:12.41 INFO  ==> MinIO is already stopped...

Where is the hacktoberfest-2022 channel?

👋 Hello, team! Ganesh here from India

hi team, I am an engineer at Gojek and we use flyte for orchestrating ML pipelines. I have a requirement to create multiple

launch plans

with different set of

fixed inputs

for an existing

workflow

. I am following flytekit documentation to fetch a workflow and create a launchplan for it. The following in the code snippet from above docs:

from flytekit import LaunchPlan

flyte_workflow = remote.fetch_workflow(
    name="my_workflow", version="v1", project="flytesnacks", domain="development"
)
launch_plan = LaunchPlan.get_or_create(name="my_launch_plan", workflow=flyte_workflow)

Creating a launchplan fails with the following error:

File "/Users/.../lib/python3.7/site-packages/flytekit/core/launch_plan.py", line 143, in create
    native_types=workflow.python_interface.inputs,
AttributeError: 'NoneType' object has no attribute 'inputs'

python_interface

attribute of workflow object returned by

fetch_workflow

is None, although the workflow is a valid one and when i register a launchplan using "register files" in cli, it works. flytekit version: 1.1.0 flyte helm chart version: 1.1.0 Any help appreciated.

Team, i am trying to modify existing registered workflow and register again with some changes but ending up with below exception. it is showing exception with launch plan already exists. kindly let me know best practices to redeploy the workflow with same launch plan.

details = "launch plan with different structure already exists with id resource_type:LAUNCH_PLAN project:"cloudops-max-flyte-demo" domain:"development" name:"snowflake_cron_scheduled_lp" version:"v1"

Hello Flyte team! We are using Flyte to schedule spark driver and executor pods. The problem is that the tolerations we use somehow don't get applied to these pods. They however work fine for all the other pods. Example tolerations on a spark pod (driver and executor the same):

Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s

Example tolerations on another pod (also flyte created).

Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
                             <http://nvidia.com/gpu=present:NoSchedule|nvidia.com/gpu=present:NoSchedule>
                             <http://nvidia.com/gpu:NoSchedule|nvidia.com/gpu:NoSchedule> op=Exists
                             purpose=compute:NoSchedule

We already tried to restart the node pool. Is this somehow an expected behavior? Flytekit version:

1.1.0