https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
ask-the-community
  • y

    Yash Panchwatkar

    10/02/2022, 12:41 PM
    hello guys .. i have recreated same setup in ubuntu and again got stuck while running flytectl demo start. this will be really helpful if anyone can help..
    s
    • 2
    • 1
  • y

    Yash Panchwatkar

    10/02/2022, 12:42 PM
    here are the logs for excited container..
  • y

    Yash Panchwatkar

    10/02/2022, 12:42 PM
    message has been deleted
    s
    p
    • 3
    • 27
  • r

    Robin Eklund

    10/03/2022, 9:10 AM
    Hi! We are looking into different alternatives how to run DBT with Flyte and came across this plugin: https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dbt Do anyone know the status of it and is it the recommended approach? The alternative we were thinking about is to run a docker image which execute the dbt command. Or maybe there are any other better way of running DBT?
    s
    k
    e
    • 4
    • 20
  • t

    Tony Vec

    10/03/2022, 1:44 PM
    This is more of a Pytest question rather than a Flyte one. I’m looking to see if there’s a preferred way of setting default values in
    SecretsManager
    within tests with the goal of not accidentally using secrets in
    /etc/secrets
    . We have a way of doing this now (I’ll add what we have in 🧵) but was hoping to see if there’s a standard way of doing this
    s
    • 2
    • 2
  • g

    George D. Torres

    10/03/2022, 6:33 PM
    What is the advice on using caching on entire workflows? I don't see any docs on that, just for tasks. My workflow doesn't return any values at the end, so I suppose I could prepend a task at the beginning of the workflow whose entire purpose is to handle the cache (i.e. have the same inputs as the workflow and add
    cache=True
    ). Is there a better way? What about for workflows that do return values?
    k
    • 2
    • 3
  • k

    Krishna Agarwal

    10/04/2022, 2:35 AM
    Hello, I have opened many PR's in UnionML repository, please anyone review those PR's. I opened the PR's on Oct 1, till now no one responded.
    s
    s
    • 3
    • 6
  • o

    Open AIMP

    10/04/2022, 4:31 AM
    Hi - I asked this question on a different channel but I believe this is the right channel for this question. Can someone help with this? https://flyte-org.slack.com/archives/C01P3B761A6/p1664854331557839
    s
    • 2
    • 1
  • s

    Samhita Alla

    10/04/2022, 11:20 AM
    @Navyansh Kesarwani, could you let us know the commands you ran?
    n
    • 2
    • 6
  • r

    Robin Eklund

    10/04/2022, 12:51 PM
    Maybe this is a stupid question. But is it possible to somehow within the workflow create dependencies between tasks without connecting them with input and output? example:
    from flytekit import task, workflow
    
    
    @task
    def task_a() -> None:
      print("task_a")
    
    
    @task
    def task_b() -> None:
      print("task_b")
    
    
    @task
    def task_c() -> None:
      print("task_c")
    
    
    @workflow
    def my_wf() -> None:
      task_a()
      task_b()
      task_c()
    this workflow would run task_a, task_b and task_c in parallel. But i want them to run sequentially.
    k
    s
    +2
    • 5
    • 17
  • g

    Geoff Salmon

    10/04/2022, 4:18 PM
    Hi, I think the enabled/disabled status of the Google Analytics feature in the console is being displayed incorrectly. I created an issue on the flyteconsole repo. Is there a process to mention it on slack or make it visible in some other way? I'm unable to label it or attach a screenshot.
    k
    s
    • 3
    • 5
  • r

    Robin Kahlow

    10/04/2022, 5:44 PM
    Hey, can Flyte handle
    field
    defaults on dataclasses? Dataclasses can't have list / dict / set defaults (https://stackoverflow.com/q/52063759) and you have to resort to something like
    str_list: List[str] = field(default_factory=lambda: ["a", "b"])
    instead, but on Flyte console when launching a workflow it just shows an empty list for me.
    k
    s
    +2
    • 5
    • 16
  • p

    Padma Priya M

    10/05/2022, 5:58 AM
    Hi, can we specify workflow version in LaunchPlan.get_or_create method.
    cron_lp = LaunchPlan.get_or_create(
        name="cron_scheduler",
        workflow=scheduler_workflow,
        schedule=CronSchedule(
            schedule="*/1 * * * *",
        ),
        default_inputs={'f': 4.5},
    )
    s
    k
    s
    • 4
    • 4
  • r

    Robert Everson

    10/05/2022, 9:45 PM
    Are there any major “gotchas” to upgrading from a much older flytepropeller/flyteplugins to a more recent version? Are there any changes required for users who register their workflows (i.e. upgrading flytekit or anything like that)?
    e
    k
    • 3
    • 14
  • h

    Hamza Azelmat

    10/06/2022, 11:58 AM
    Hey 👋🏻 , Has anyone made the flyte deck work for notebook tasks? I got a 404 after a successful run Also, is there a way to copy the executed notebook ipynb or html to GCS ? I got file not found when I tried to use the implicit output of notebook task
    No such file or directory: '/v3/f25eaabfb23924c25946-n1-0/d1a84d1c0cea75f237d21a70dcde6810/model1-out.ipynb'
    k
    c
    • 3
    • 25
  • i

    Ian Calvert

    10/06/2022, 3:50 PM
    Hi, trying to plan out the developer experience for people working in a flyte setup where we'd be making heavier use of pre-registered workflows/tasks (reference_task decorator). That pushes it towards using local k8s which is fine but my first experiments suggest it's pretty slow, adding something like 10+s per task. Any guide as to why that might be?
    y
    d
    • 3
    • 18
  • l

    Louis DiNatale

    10/06/2022, 5:18 PM
    Hi everyone! I was wondering where i could look inside k8s to verify that a resource quota had been updated?
    k
    • 2
    • 7
  • f

    Frank Shen

    10/06/2022, 6:35 PM
    Hello, I have a flyte .py file that imports a module I created from a subdir called lib and it contains another workflow that is intended as subflow.
    from lib.subwf import my_subwf
    I can run the flyte flow successfully in local mode. However, it errored when running with --remote in the demo flyte cluster I started on my PC. The error msg is:
    ModuleNotFoundError: No module named 'lib'
    k
    • 2
    • 4
  • f

    Frank Shen

    10/06/2022, 6:36 PM
    I am not able to find flyte documentation for the resolution of this issue. Could anyone share the root cause of the error and the fix?
    n
    k
    y
    • 4
    • 26
  • f

    Frank Shen

    10/06/2022, 6:36 PM
    Thank you!
  • a

    Adwitiya

    10/07/2022, 1:21 PM
    How can I register my code( written using flytekit ) in flyte sandbox and run it from remote (executing wf from a python code)
    s
    s
    k
    • 4
    • 7
  • s

    Srishti Mehra

    10/07/2022, 3:21 PM
    I wrote the following TypeTransformer for Light GBM Dataset and am seeing it default to pickle on registering the tasks and workflow that use a Light GBM Dataset. Would any be able to help point me in the right direction to troubleshoot this?
    class LGBMDatasetTransformer(extend.TypeTransformer[lgb.Dataset]):
        _TYPE_INFO = flytekit.BlobType(
            format="binary", dimensionality=flytekit.BlobType.BlobDimensionality.MULTIPART
        )
    
        def __init__(self) -> None:
            super(LGBMDatasetTransformer, self).__init__(name="lgbdataset-transform", t=lgb.Dataset)
    
        def get_literal_type(self, t: Type[lgb.Dataset]) -> flytekit.LiteralType:
            return flytekit.LiteralType(blob=self._TYPE_INFO)
    
        def to_literal(
            self,
            ctx: flytekit.FlyteContext,
            python_val: lgb.Dataset,
            python_type: Type[lgb.Dataset],
            expected: flytekit.LiteralType,
        ) -> flytekit.Literal:
            """
            Convert from the given python type object ``lgb.Dataset`` to the Literal representation.
            """
            # Step 1: Upload all the data into a remote place recommended by Flyte
    
            local_file = ctx.file_access.get_random_local_path() + ".data"
            lgb.Dataset.save_binary(python_val, local_file)
    
            remote_dir = ctx.file_access.get_random_remote_directory()
            ctx.file_access.upload_directory(local_file, remote_dir)
            # Step 2: Return a pointer to this remote_dir in the form of a Literal
            return flytekit.Literal(
                scalar=flytekit.Scalar(
                    blob=flytekit.Blob(
                        uri=remote_dir, metadata=flytekit.BlobMetadata(type=self._TYPE_INFO)
                    )
                )
            )
    
        def to_python_value(
            self,
            ctx: flytekit.FlyteContext,
            lv: flytekit.Literal,
            expected_python_type: Type[lgb.Dataset],
        ) -> lgb.Dataset:
            """
            In this method, we want to be able to re-hydrate the custom object from Flyte Literal value.
            """
            # Step 1: Download remote data locally
            local_dir = ctx.file_access.get_random_local_directory()
            ctx.file_access.download_directory(lv.scalar.blob.uri, local_dir)
            # Step 2: Create the ``lgb.Dataset`` object
            return lgb.Dataset.construct(lgb.Dataset(local_dir))
    
    
    extend.TypeEngine.register(LGBMDatasetTransformer())
    k
    r
    k
    • 4
    • 19
  • f

    fei sun

    10/08/2022, 3:10 AM
    Hi all, the task run log is external link and flyte doesn’t collect and show the log real time on the webui, also flyte doesn’t persist the log. the log is gone after the pod destruction, any idea how to handle this?
    k
    • 2
    • 2
  • y

    Yash Panchwatkar

    10/08/2022, 7:13 AM
    hello guys .. i am trying to start sandbox environment in k8s cluster but minio pod is goes in crashloopbackoff again and again could you please help me to find what's getting wrong.. here.
  • y

    Yash Panchwatkar

    10/08/2022, 7:15 AM
    result from describe command for minio
    Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
    Events:
      Type     Reason     Age                 From               Message
      ----     ------     ----                ----               -------
      Normal   Scheduled  16m                 default-scheduler  Successfully assigned flyte/minio-5745464bb-zthw8 to ip-172-31-31-189.ap-south-1.compute.internal
      Normal   Pulling    16m                 kubelet            Pulling image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>"
      Normal   Pulled     16m                 kubelet            Successfully pulled image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" in 13.816089274s
      Normal   Created    14m (x5 over 16m)   kubelet            Created container minio
      Normal   Started    14m (x5 over 16m)   kubelet            Started container minio
      Normal   Pulled     14m (x4 over 16m)   kubelet            Container image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" already present on machine
      Warning  BackOff    85s (x61 over 15m)  kubelet            Back-off restarting failed container
    and result from logs
    [root@ip-172-31-31-189 ~]# kubectl logs  minio-5745464bb-zthw8 -n flyte
     07:15:01.89
     07:15:01.90 Welcome to the Bitnami minio container
     07:15:01.90 Subscribe to project updates by watching <https://github.com/bitnami/bitnami-docker-minio>
     07:15:01.90 Submit issues and feature requests at <https://github.com/bitnami/bitnami-docker-minio/issues>
     07:15:01.90
     07:15:01.90 INFO  ==> ** Starting MinIO setup **
     07:15:02.01 INFO  ==> Starting MinIO in background...
     07:15:12.01 INFO  ==> Adding local Minio host to 'mc' configuration...
     07:15:12.41 INFO  ==> MinIO is already stopped...
  • u

    0xsapphir3

    10/09/2022, 8:43 AM
    Where is the hacktoberfest-2022 channel?
  • k

    Ketan (kumare3)

    10/09/2022, 3:00 PM
    #hacktoberfest-2022
  • g

    Ganesh Patil

    10/10/2022, 3:13 AM
    👋 Hello, team! Ganesh here from India
    k
    • 2
    • 1
  • k

    Keshav Sharma

    10/10/2022, 4:43 AM
    hi team, I am an engineer at Gojek and we use flyte for orchestrating ML pipelines. I have a requirement to create multiple
    launch plans
    with different set of
    fixed inputs
    for an existing
    workflow
    . I am following flytekit documentation to fetch a workflow and create a launchplan for it. The following in the code snippet from above docs:
    from flytekit import LaunchPlan
    
    flyte_workflow = remote.fetch_workflow(
        name="my_workflow", version="v1", project="flytesnacks", domain="development"
    )
    launch_plan = LaunchPlan.get_or_create(name="my_launch_plan", workflow=flyte_workflow)
    Creating a launchplan fails with the following error:
    File "/Users/.../lib/python3.7/site-packages/flytekit/core/launch_plan.py", line 143, in create
        native_types=workflow.python_interface.inputs,
    AttributeError: 'NoneType' object has no attribute 'inputs'
    python_interface
    attribute of workflow object returned by
    fetch_workflow
    is None, although the workflow is a valid one and when i register a launchplan using "register files" in cli, it works. flytekit version: 1.1.0 flyte helm chart version: 1.1.0 Any help appreciated.
    k
    s
    k
    • 4
    • 13
  • s

    Sathish kumar Venkatesan

    10/10/2022, 8:41 AM
    Team, i am trying to modify existing registered workflow and register again with some changes but ending up with below exception. it is showing exception with launch plan already exists. kindly let me know best practices to redeploy the workflow with same launch plan.
    details = "launch plan with different structure already exists with id resource_type:LAUNCH_PLAN project:"cloudops-max-flyte-demo" domain:"development" name:"snowflake_cron_scheduled_lp" version:"v1"
    s
    k
    • 3
    • 6
Powered by Linen
Title
s

Sathish kumar Venkatesan

10/10/2022, 8:41 AM
Team, i am trying to modify existing registered workflow and register again with some changes but ending up with below exception. it is showing exception with launch plan already exists. kindly let me know best practices to redeploy the workflow with same launch plan.
details = "launch plan with different structure already exists with id resource_type:LAUNCH_PLAN project:"cloudops-max-flyte-demo" domain:"development" name:"snowflake_cron_scheduled_lp" version:"v1"
s

Samhita Alla

10/10/2022, 10:49 AM
@Sathish kumar Venkatesan, can you give a different version?
s

Sathish kumar Venkatesan

10/10/2022, 10:51 AM
@Samhita Alla i can try with different version. but my goal is to use same version and same launch plan. since i am trying to register via cicd pipeline.
s

Samhita Alla

10/10/2022, 10:53 AM
If you’re modifying the code, you can’t use the same version to register again.
s

Sathish kumar Venkatesan

10/10/2022, 11:20 AM
is there anyway to remove previously registered workflow ?
s

Samhita Alla

10/10/2022, 12:09 PM
Um no. You cannot delete a registered workflow; you can archive it, though.
k

Ketan (kumare3)

10/10/2022, 1:50 PM
Archive
View count: 2