https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
ask-the-community
  • r

    Rahul Mehta

    09/22/2022, 1:57 AM
    I might be missing something, but is there a way to pass multiple arguments to a map task (ie. if I have a list
    [1,2,3]
    , can I pass each item to a map task along with some other fixed inputs that are common across all tasks). All the examples in the docs only include tasks w/ a single argument Relatedly, is there a way to use a workflow as part of a map task (ie. map a sub-workflow over a sequence of inputs) Edit: guess not
    Map tasks only accept python function tasks with 0 or 1 inputs
    s
    k
    k
    • 4
    • 13
  • s

    Sanjay Chouhan

    09/22/2022, 6:38 AM
    Hi, I had setup Flyte Sandbox on EKS by following these steps https://docs.flyte.org/en/latest/deployment/sandbox.html#deploy-flyte-sandbox-to-a-cloud-kubernetes-cluster I can access the console UI. But when I run "`pyflyte --config .flyte/config.yaml run --remote example.py wf --n 500 --mean 42 --sigma 2`" I get error like " flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmp9l0dj7e1/script_mode.tar.gz to http://localhost:30084/my-s3-bucket --- Original exception: HTTPConnectionPool(host='localhost', port=30084): Max retries exceeded with url: /my-s3-bucket/bh/flytesnacks/development/OWEO " I have mentioned the details in config.yaml like,
    admin:
    # For GRPC endpoints you might want to use dns:///flyte.myexample.com
    endpoint: dns:///a4ad903c61##################.<http://us-west-1.elb.amazonaws.com:80|us-west-1.elb.amazonaws.com:80>
    authType: Pkce
    insecure: true
    logger:
    show-source: true
    level: 0
    storage:
    connection:
    access-key: minio
    auth-type: accesskey
    disable-ssl: true
    endpoint: <http://a093eb##############.us-west-1.elb.amazonaws.com:9001/>
    region: us-east-1
    secret-key: miniostorage
    type: minio
    container: "my-s3-bucket"
    enable-multicontainer: true
    Why it's checking in localhost when I have mentioned the AWS ingress URL?
    s
    k
    • 3
    • 15
  • s

    Sathish kumar Venkatesan

    09/22/2022, 10:51 AM
    Team I am running below simple snowflake query, however it take more than hours to execute. i could see Attempt 01 queued for long and then execute,
    from flytekit import kwtypes, workflow
    from flytekitplugins.snowflake import SnowflakeConfig, SnowflakeTask
    
    
    snowflake_task_no_io = SnowflakeTask(
        name="sql.snowflake.no_io",
        inputs={},
        query_template="select * from TEST_DEV.IDENTITY.TEST_123 ;",
        output_schema_type=None,
        task_config=SnowflakeConfig(
            account="xxxxx.us-east-1",
            database="TEST_DEV",
            schema="IDENTITY",
            warehouse="DEMO_WH",
        ),
    )
    
    
    @workflow
    def no_io_wf():
        return snowflake_task_no_io()
    s
    k
    k
    • 4
    • 37
  • k

    KS Tarun

    09/22/2022, 12:08 PM
    Hi, How to change the container image name in the task decorator to fetch it directly from the sandbox.config file ? I've tried as mentioned below:
    @task(container_image="{{.image.trainer.fqn }}:{{.image.trainer.version}}")
    My sandbox.config looks like this:
    [images]
    trainer = <http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest>
    predictor = <http://ghcr.io/flyteorg/flytecookbook:pima_diabetes-d4838f0f5e39a21f845a93b9e3375a675bd75eaa|ghcr.io/flyteorg/flytecookbook:pima_diabetes-d4838f0f5e39a21f845a93b9e3375a675bd75eaa>
    I'm getting this error:
    raise AssertionError(f"Image Config with name {name} not hound in the configuration")
    AssertionError:Image Config with name trainer not found in the configuration
    Any suggestions about this ?
    j
    e
    +6
    • 9
    • 56
  • k

    Katrina P

    09/22/2022, 9:02 PM
    Hey folks! Curious if your Flyte users have a specific folder structure to hold generalized tasks, workflows, etc? We're brainstorming what might be the best way to organize and isolate workflow dependencies. For example, some workflows could share the same Dockerfile & requirements, but others may have their own... etc... any best practices folks developed at their companies they're willing to share? We tried to follow something similar to Flytesnacks but I think it got a bit disorganized.
    y
    k
    r
    • 4
    • 12
  • s

    seunggs

    09/23/2022, 12:24 AM
    Been struggling with s3 PutObject access denied issue for days now (@Yee I’m posting a new one here because the existing Slack thread won’t load for me for some reason). Any idea why having a service account with the
    flyte-user-role
    (which has full s3 access) attached as an annotation still gives me PutObject access denied error?
    k
    • 2
    • 22
  • s

    seunggs

    09/23/2022, 12:25 AM
    This is blocking me from running even a simple test workflow on Flyte in a production cluster
  • s

    seunggs

    09/23/2022, 12:25 AM
    Would really appreciate some guidance
  • r

    Robin Kahlow

    09/23/2022, 11:23 AM
    Hey, I have a workflow that uses dataclass parameters as inputs. In Flyte console when launching the workflow, it shows the correct parameters (left side first screenshot). But when inspecting the inputs of both the task (right side first screenshot) and the workflow (second screenshot) it shows an older version of the DataConfig dataclass with different parameters (which are using defaults). UI Version 1.1.7, Admin Version 1.1.30 // Ah I think it's because I relaunched from an older version selecting my new version, so it still had those old parameters but they were hidden or something. Launching a new workflow from scratch works fine.
    k
    • 2
    • 1
  • r

    Robin Kahlow

    09/23/2022, 11:37 AM
    Also entering empty string on a string argument on those dataclasses makes the argument disappear from the input (instead of
    arg: ""
    ), rather than storing an empty string. That causes it to use the default of the field rather than using an empty string.
    k
    • 2
    • 3
  • r

    Rupsha Chaudhuri

    09/23/2022, 4:41 PM
    Hi team… I’m seeing this error in a slightly long running flyte task, which results in the task getting triggered again and again.
    object [my_domainctw5x2vdnkvd-n0-0] terminated in the background, manually
    d
    k
    • 3
    • 29
  • a

    Ankit Goyal

    09/23/2022, 8:07 PM
    This is the sub workflow DAG:
    k
    j
    • 3
    • 13
  • s

    seunggs

    09/23/2022, 10:38 PM
    Hi, is there a limit to the length of the workflow name? I’m trying to launch an execution via rest api, and getting this 400 error:
    error: "size of name exceeded length 20 : my-project.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1"
    . What’s strange is that this seems to run fine from the flyte dashboard ui?
  • s

    seunggs

    09/23/2022, 10:40 PM
    20 seems really short for a full workflow path?
  • s

    seunggs

    09/23/2022, 10:43 PM
    This is my payload to `POST /executions`:
  • s

    seunggs

    09/23/2022, 10:44 PM
    {
      project: "shelly-robotics-bidepal-robot",
      domain: "development",
      name: "my-project.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1",
      spec: {
        launch_plan: {
          project: "shelly-robotics-bidepal-robot",
          domain: "development",
          name: "my-project.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1",
          version: "0.0.1",
        },
        inputs: {
          x: 1,
        },
      },
    };
    y
    k
    • 3
    • 79
  • m

    Maarten T

    09/26/2022, 8:12 AM
    Hi all, I’ve been working with Flyte the couple last weeks to implement some model workflows. I’m struggling with passing files to a containertask. Is it possible to pass a list of files or directory? I managed to pass 1 FlyteFile, but the model requires multiple files to run. If I pass a List here to the inputs, the copilot-downloader can’t load the files.
    floodcast_model = ContainerTask(
        name="floodcast-docker",
        input_data_dir="/input",
        output_data_dir="/output",
        inputs=kwtypes(rainfall=List[FlyteFile]),
        outputs=kwtypes(floodmaps=List[FlyteFile]),
        image="<myimage>",
        command=["/env/bin/python", "run.py"])
    k
    • 2
    • 10
  • s

    Sebastian

    09/26/2022, 3:03 PM
    Hello, I want to archive all current launch plans for a workflow and register a new launch plan in CI/CD where I only know the current version (gitsha). How can I do this? Basically I would like
    flytectl update launchplan ... -- version $GITSHA --activate >> --archive-old <<
    . This seems like a basic use case but I have not found any resources on it. Please let me know if I overlooked anything
    k
    y
    • 3
    • 5
  • r

    Robert Everson

    09/26/2022, 7:20 PM
    Hi all, has anyone ever seen a
    EventSinkError
    that on some workflows appears to occur occasionally, but on others happens all the time? We moved to a new k8s cluster for our flyteadmin instance (and from 3 instances to 1) and the timing of that seems to mostly line up with this error starting, but after checking the flyteadmin logs, I can see propeller sending a
    CreateWorkflowEvent
    and it getting successfully registered, but propeller will still throw an
    EventRecordingFailed: failed to record node event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: protocol error]
    k
    • 2
    • 3
  • s

    Sebastian

    09/27/2022, 5:42 AM
    Hi all, consider a workflow
    launch_plan.LaunchPlan.get_or_create(
        workflow=wf,
        name="my_wf_prod", # prod specific
        schedule=CronSchedule(schedule="1 0 * * *"),
        default_inputs={
            "execution_date": datetime.now(), # aside: this does not work. How do I provide an execution date?
        },
        fixed_inputs={
            "out_path": consts.OUT_S3_PATH_PROD # prod specific vars
        },
    )
    there is also a copy of this launch plan for the dev env, so that I can plug in the dev env vars. But this is silly and requires two copies. How can I parameterize workflows with something like env vars so I can vary them between prod and dev?
    k
    k
    • 3
    • 5
  • f

    fei sun

    09/27/2022, 6:51 AM
    Problem solved. hi all, I am following `https://docs.flyte.org/projects/cookbook/en/latest/userguide_setup.html`to demo flyte, after
    flytectl demo start
    finished successfully, it’s able to open flyte webui which is localhost:30080, but pyflyte remote run failed with the following error message:
    fei.sun@MacBook-Pro cookbook % pyflyte run --remote core/flyte_basics/hello_world.py my_wf
    
    
    E0927 14:46:18.763713000 4366484864 <http://ssl_transport_security.cc:1495]|ssl_transport_security.cc:1495]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
    E0927 14:46:18.765990000 4366484864 <http://ssl_transport_security.cc:1495]|ssl_transport_security.cc:1495]>    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
    {"asctime": "2022-09-27 14:46:18,781", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1664261178.780964000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3261,\"referenced_errors\":[{\"created\":\"@1664261178.780963000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":167,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
    {"asctime": "2022-09-27 14:46:18,986", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1664261178.985906000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3261,\"referenced_errors\":[{\"created\":\"@1664261178.985905000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":167,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
    Traceback (most recent call last):
      File "/Users/fei.sun/Library/Python/3.8/bin//pyflyte", line 8, in <module>
        sys.exit(main())
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/flytekit/clis/sdk_in_container/run.py", line 539, in _run
        remote_entity = remote.register_script(
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/flytekit/remote/remote.py", line 596, in register_script
        upload_location, md5_bytes = fast_register_single_script(
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/flytekit/tools/script_mode.py", line 113, in fast_register_single_script
        upload_location = create_upload_location_fn(content_md5=md5)
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
        return super(SynchronousFlyteClient, self).create_upload_location(
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/flytekit/clients/raw.py", line 41, in handler
        return fn(*args, **kwargs)
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/flytekit/clients/raw.py", line 854, in create_upload_location
        return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/grpc/_channel.py", line 946, in __call__
        return _end_unary_response_blocking(state, call, False, None)
      File "/Users/fei.sun/Library/Python/3.8/lib/python/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
        raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNAVAILABLE
    	details = "failed to connect to all addresses"
    	debug_error_string = "{"created":"@1664261179.392327000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3261,"referenced_errors":[{"created":"@1664261179.392310000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
    any idea how to fix this?
  • f

    fei sun

    09/27/2022, 6:55 AM
    also, flyte limits the max recursion depth, what’s the max depth number?
    s
    k
    m
    • 4
    • 8
  • t

    Tamis van der Laan

    09/27/2022, 8:39 AM
    https://github.com/flyteorg/flyte/issues/2884
  • t

    Tamis van der Laan

    09/27/2022, 8:40 AM
    @fei sun same bug as this onre
    f
    • 2
    • 1
  • t

    Tamis van der Laan

    09/27/2022, 8:42 AM
    I'm running flyte remotely and using the mlflow python module. Locally it works but remove it cant find the module. How does this work? Do I neede to install this remotely?
    s
    k
    • 3
    • 2
  • s

    Samhita Alla

    09/27/2022, 11:57 AM
    cc: @Prafulla Mahindrakar
    p
    k
    y
    • 4
    • 7
  • h

    Hampus Rosvall

    09/28/2022, 6:17 AM
    Hey, when I try to look at some launch plans I get the following error in the console UI, would you know what could cause it and how to fix it?
    s
    • 2
    • 7
  • a

    Anthony

    09/28/2022, 8:31 AM
    Hi all 👋 I have a problem of sharing data between tasks. I found a similar issue here in discussions (link)
    Workflow[flyte-anti-fraud-ml:development:app.workflow.main_flow] failed. RuntimeExecutionError: max number of system retry attempts [31/30] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: error file @[<s3://my-s3-bucket/metadata/propeller/flyte-anti-fraud-ml-development-f31c365f02c114639b00/n0/data/0/error.pb>] is too large [28775519] bytes, max allowed [10485760] bytes
    I added
    max-output-size-bytes
    params to the
    flyte-propeller-config
    and wait to apply all changes before re-submitting a new task.
    kubectl edit configmap -n flyte flyte-propeller-config
    My propeller section of
    flyte-propeller-config
    looks like:
    core.yaml: |
        manager:
          pod-application: flytepropeller
          pod-template-container-name: flytepropeller
          pod-template-name: flytepropeller-template
        propeller:
          max-output-size-bytes: 52428800
          downstream-eval-duration: 30s
          enable-admin-launcher: true
          leader-election:
            enabled: true
            lease-duration: 15s
            lock-config-map:
              name: propeller-leader
              namespace: flyte
            renew-deadline: 10s
            retry-period: 2s
          limit-namespace: all
          max-workflow-retries: 3
          metadata-prefix: metadata/propeller
          metrics-prefix: flyte
          prof-port: 10254
    Task configuration has been set up via
    kubectl -n flyte edit cm flyte-admin-base-config
    storage.yaml: |
        storage:
          type: minio
          container: "my-s3-bucket"
          stow:
            kind: s3
            config:
              access_key_id: minio
              auth_type: accesskey
              secret_key: miniostorage
              disable_ssl: true
              endpoint: <http://minio.flyte.svc.cluster.local:9000>
              region: us-east-1
          signedUrl:
            stowConfigOverride:
              endpoint: <http://localhost:30084>
          enable-multicontainer: false
          limits:
            maxDownloadMBs: 50
      task_resource_defaults.yaml: |
        task_resources:
          defaults:
            cpu: 1
            memory: 3000Mi
            storage: 100Mi
          limits:
            cpu: 4
            gpu: 1
            memory: 3Gi
            storage: 500Mi
    Also changing
    maxDownloadMBs
    didn’t change the situation Changing cache
    max_size_mbs
    in
    flyte-propeller-config
    from 0 to some custom value also not working:
    cache.yaml: |
        cache:
          max_size_mbs: 100
          target_gc_percent: 70
    I ty to change different time with different params but the error was arising during each new executions. I saw that none of
    max-output-size-bytes
    or
    max-workflow-retries
    (changed from 30 --> 3) are passed to the workflow execution:
    RuntimeExecutionError: max number of system retry attempts [31/30] exhausted...
    
    error file @[<s3://my-s3-bucket/metadata/propeller/flyte-anti-fraud-ml-development-f31c365f02c114639b00/n0/data/0/error.pb>] is too large [28775519] bytes, max allowed [10485760] bytes...
    Hereafter are my cli steps to create a new execution:
    - kubectl -n flyte edit cm flyte-admin-base-config
    - kubectl edit configmap -n flyte flyte-propeller-config
    - flytectl get task-resource-attribute -p flyteexamples -d development
    - flytectl update project -p flyte-anti-fraud-ml -d development --storage.cache.max_size_mbs 100
    - flytectl get launchplan --project flyte-anti-fraud-ml --domain development app.workflow.main_flow --latest --execFile exec_spec.yaml
    - flytectl create execution --project flyte-anti-fraud-ml --domain development --execFile exec_spec.yaml
    What additional steps I have to do to force flytectl to use my propeller changes and solve the problem of a max 10Mb size allowed for serialized uploads to flyte?
    s
    • 2
    • 9
  • s

    Sanjay Chouhan

    09/28/2022, 9:33 AM
    https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_register_files.html How to generate the protobuf files for a task or workflow for registrations?
    a
    s
    k
    • 4
    • 5
  • p

    Padma Priya M

    09/28/2022, 9:36 AM
    Hi I am trying to integrate jupyter notebook with flyte. When I execute the script in local I am able to get the output but when I execute in remote I am getting error.
    [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
    [ff43e19ed1617402a8c6-n0-0] terminated with exit code (1). Reason [Error]. Message: 
    ine 286, in _handle_annotated_task
        _dispatch_execute(ctx, task_def, inputs, output_prefix)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 76, in _dispatch_execute
        logger.debug(f"Starting _dispatch_execute for {task_def.name}")
    AttributeError: 'function' object has no attribute 'name'
    Traceback (most recent call last):
      File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
        sys.exit(fast_execute_task_cmd())
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in _call_
        return self.main(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 502, in fast_execute_task_cmd
        subprocess.run(cmd, check=True)
      File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ff43e19ed1617402a8c6/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ff43e19ed1617402a8c6/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/qf/ff43e19ed1617402a8c6-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/qf/ff43e19ed1617402a8c6-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/72/flytesnacks/development/LXFFQS6IOSM2OAH57AOZKM2SZA======/scriptmode.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'flytekitplugins.papermill.task', 'task-name', '_dummy_task_func']' returned non-zero exit status 1.
    k
    s
    • 3
    • 21
Powered by Linen
Title
p

Padma Priya M

09/28/2022, 9:36 AM
Hi I am trying to integrate jupyter notebook with flyte. When I execute the script in local I am able to get the output but when I execute in remote I am getting error.
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[ff43e19ed1617402a8c6-n0-0] terminated with exit code (1). Reason [Error]. Message: 
ine 286, in _handle_annotated_task
    _dispatch_execute(ctx, task_def, inputs, output_prefix)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 76, in _dispatch_execute
    logger.debug(f"Starting _dispatch_execute for {task_def.name}")
AttributeError: 'function' object has no attribute 'name'
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in _call_
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 502, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ff43e19ed1617402a8c6/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ff43e19ed1617402a8c6/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/qf/ff43e19ed1617402a8c6-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/qf/ff43e19ed1617402a8c6-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/72/flytesnacks/development/LXFFQS6IOSM2OAH57AOZKM2SZA======/scriptmode.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'flytekitplugins.papermill.task', 'task-name', '_dummy_task_func']' returned non-zero exit status 1.
pyflyte run --remote --image ghcr.io/flyteorg/flytecookbook:papermilltasks-latest example.py nb_to_python_wf --f 2.3 command used to execute.
import math
import os
import sys
import pathlib

from flytekit import kwtypes, task, workflow
from flytekitplugins.papermill import NotebookTask

nb = NotebookTask(
    name="simple-nb",
    notebook_path=os.path.join(
        sys.path[0], "notebook_sample.ipynb"
    ),
    render_deck=True,
    inputs=kwtypes(v=float),
    outputs=kwtypes(square=float),
)


@task
def square_root_task(f: float) -> float:
    return math.sqrt(f)


@workflow
def nb_to_python_wf(f: float) -> float:
    out = nb(v=f)
    return square_root_task(f=out.square)

if __name__ == "__main__":
    print(nb_to_python_wf(f=3.14))
k

Kevin Su

09/28/2022, 11:19 AM
which flytekit version you are using?
p

Padma Priya M

09/28/2022, 11:25 AM
using flytekit 1.1.3
s

Samhita Alla

09/29/2022, 12:48 PM
This seems like a bug.
@Kevin Su, any idea why this might have cropped up?
k

Kevin Su

09/29/2022, 3:12 PM
yes, this is bug now in 1.1.3. try to downgrade to 1.1.1. I’ll create pr to fix it shortly. btw, currently pyflyte run will only upload the workflow code (example.py) to s3, and will not upload notebook_sample.ipynb. Therefore, you should use pyflyte register instead, and manually trigger workflow by yourself.
pyflyte register --image <http://ghcr.io/flyteorg/flytecookbook:papermilltasks-latest|ghcr.io/flyteorg/flytecookbook:papermilltasks-latest> example.py
s

Samhita Alla

10/05/2022, 1:42 PM
Have you installed the plugin?
Please install all the requirements in your local environment.
Can you try running the example with the latest flytekit version — 1.2.0? Also, this error is related to not installing the papermill plugin. I’d recommend you to create a new virtual env, update the dependencies (you can copy the requirements from flytesnacks papermill directory and update them using
pip-compile <http://requirements.in|requirements.in>
), install the dependencies, and run the pyflyte register again.
p

Padma Priya M

10/19/2022, 4:23 AM
Hi, I installed all the dependencies that were mentioned in flytesnacks papermill directory with latest flytekit version. I was able to register the workflow but when I created an execution, I am getting error.
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile <http://requirements.in|requirements.in>
#
--index-url <https://sonatype.adfdata.net/repository/pypi_adf/simple>
--extra-index-url <https://pypi.org/simple>
--trusted-host <http://sonatype.adfdata.net|sonatype.adfdata.net>

ansiwrap==0.8.4
    # via papermill
arrow==1.2.2
    # via jinja2-time
asttokens==2.0.5
    # via stack-data
attrs==21.4.0
    # via jsonschema
backcall==0.2.0
    # via ipython
beautifulsoup4==4.11.1
    # via nbconvert
binaryornot==0.4.4
    # via cookiecutter
bleach==5.0.0
    # via nbconvert
certifi==2021.10.8
    # via requests
cffi==1.15.1
    # via cryptography
chardet==4.0.0
    # via binaryornot
charset-normalizer==2.0.12
    # via requests
checksumdir==1.2.0
    # via flytekit
click==8.1.2
    # via
    #   cookiecutter
    #   flytekit
    #   papermill
cloudpickle==2.0.0
    # via flytekit
cookiecutter==1.7.3
    # via flytekit
croniter==1.3.4
    # via flytekit
cryptography==38.0.1
    # via secretstorage
dataclasses-json==0.5.7
    # via flytekit
debugpy==1.6.0
    # via ipykernel
decorator==5.1.1
    # via
    #   ipython
    #   retry
defusedxml==0.7.1
    # via nbconvert
deprecated==1.2.13
    # via flytekit
diskcache==5.4.0
    # via flytekit
docker-image-py==0.1.12
    # via flytekit
docstring-parser==0.13
    # via flytekit
entrypoints==0.4
    # via
    #   jupyter-client
    #   nbconvert
    #   papermill
executing==0.8.3
    # via stack-data
fastjsonschema==2.15.3
    # via nbformat
flyteidl==0.24.21
    # via flytekit
flytekit==0.32.3
    # via flytekitplugins-papermill
flytekitplugins-papermill==0.32.3
    # via -r <http://requirements.in|requirements.in>
googleapis-common-protos==1.56.0
    # via
    #   flyteidl
    #   grpcio-status
grpcio==1.44.0
    # via
    #   flytekit
    #   grpcio-status
grpcio-status==1.44.0
    # via flytekit
idna==3.3
    # via requests
importlib-metadata==4.11.3
    # via keyring
ipykernel==6.13.0
    # via flytekitplugins-papermill
ipython==8.2.0
    # via ipykernel
jedi==0.18.1
    # via ipython
jeepney==0.8.0
    # via
    #   keyring
    #   secretstorage
jinja2==3.1.1
    # via
    #   cookiecutter
    #   jinja2-time
    #   nbconvert
jinja2-time==0.2.0
    # via cookiecutter
jsonschema==4.4.0
    # via nbformat
jupyter-client==7.2.2
    # via
    #   ipykernel
    #   nbclient
jupyter-core==4.9.2
    # via
    #   jupyter-client
    #   nbconvert
    #   nbformat
jupyterlab-pygments==0.2.1
    # via nbconvert
keyring==23.5.0
    # via flytekit
markupsafe==2.1.1
    # via
    #   jinja2
    #   nbconvert
marshmallow==3.15.0
    # via
    #   dataclasses-json
    #   marshmallow-enum
    #   marshmallow-jsonschema
marshmallow-enum==1.5.1
    # via dataclasses-json
marshmallow-jsonschema==0.13.0
    # via flytekit
matplotlib-inline==0.1.3
    # via
    #   ipykernel
    #   ipython
mistune==0.8.4
    # via nbconvert
mypy-extensions==0.4.3
    # via typing-inspect
natsort==8.1.0
    # via flytekit
nbclient==0.6.0
    # via
    #   nbconvert
    #   papermill
nbconvert==6.5.0
    # via flytekitplugins-papermill
nbformat==5.3.0
    # via
    #   nbclient
    #   nbconvert
    #   papermill
nest-asyncio==1.5.5
    # via
    #   ipykernel
    #   jupyter-client
    #   nbclient
numpy==1.22.3
    # via
    #   pandas
    #   pyarrow
packaging==21.3
    # via
    #   ipykernel
    #   marshmallow
    #   nbconvert
pandas==1.4.2
    # via flytekit
pandocfilters==1.5.0
    # via nbconvert
papermill==2.3.4
    # via flytekitplugins-papermill
parso==0.8.3
    # via jedi
pexpect==4.8.0
    # via ipython
pickleshare==0.7.5
    # via ipython
poyo==0.5.0
    # via cookiecutter
prompt-toolkit==3.0.29
    # via ipython
protobuf==3.20.0
    # via
    #   flyteidl
    #   flytekit
    #   googleapis-common-protos
    #   grpcio-status
    #   protoc-gen-swagger
protoc-gen-swagger==0.1.0
    # via flyteidl
psutil==5.9.0
    # via ipykernel
ptyprocess==0.7.0
    # via pexpect
pure-eval==0.2.2
    # via stack-data
py==1.11.0
    # via retry
pyarrow==6.0.1
    # via flytekit
pycparser==2.21
    # via cffi
pygments==2.11.2
    # via
    #   ipython
    #   nbconvert
pyparsing==3.0.8
    # via packaging
pyrsistent==0.18.1
    # via jsonschema
python-dateutil==2.8.2
    # via
    #   arrow
    #   croniter
    #   flytekit
    #   jupyter-client
    #   pandas
python-json-logger==2.0.2
    # via flytekit
python-slugify==6.1.1
    # via cookiecutter
pytimeparse==1.1.8
    # via flytekit
pytz==2022.1
    # via
    #   flytekit
    #   pandas
pyyaml==6.0
    # via
    #   flytekit
    #   papermill
pyzmq==22.3.0
    # via jupyter-client
regex==2022.3.15
    # via docker-image-py
requests==2.27.1
    # via
    #   cookiecutter
    #   flytekit
    #   papermill
    #   responses
responses==0.20.0
    # via flytekit
retry==0.9.2
    # via flytekit
secretstorage==3.3.3
    # via keyring
six==1.16.0
    # via
    #   asttokens
    #   bleach
    #   cookiecutter
    #   grpcio
    #   python-dateutil
sortedcontainers==2.4.0
    # via flytekit
soupsieve==2.3.2
    # via beautifulsoup4
stack-data==0.2.0
    # via ipython
statsd==3.3.0
    # via flytekit
tenacity==8.0.1
    # via papermill
text-unidecode==1.3
    # via python-slugify
textwrap3==0.9.2
    # via ansiwrap
tinycss2==1.1.1
    # via nbconvert
tornado==6.1
    # via
    #   ipykernel
    #   jupyter-client
tqdm==4.64.0
    # via papermill
traitlets==5.1.1
    # via
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
typing-extensions==4.1.1
    # via
    #   flytekit
    #   typing-inspect
typing-inspect==0.7.1
    # via dataclasses-json
urllib3==1.26.9
    # via
    #   flytekit
    #   requests
    #   responses
wcwidth==0.2.5
    # via prompt-toolkit
webencodings==0.5.1
    # via
    #   bleach
    #   tinycss2
wheel==0.37.1
    # via flytekit
wrapt==1.14.0
    # via
    #   deprecated
    #   flytekit
zipp==3.8.0
    # via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
# setuptools
@Samhita Alla @Kevin Su
s

Samhita Alla

10/19/2022, 11:43 AM
Your flytekit version is 0.32.3; it should be the latest.
Let me share with you the requirements.txt file.
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    /Library/Developer/CommandLineTools/usr/bin/make requirements.txt
#
ansiwrap==0.8.4
    # via papermill
appnope==0.1.3
    # via
    #   ipykernel
    #   ipython
arrow==1.2.3
    # via jinja2-time
asttokens==2.0.8
    # via stack-data
attrs==22.1.0
    # via
    #   jsonschema
    #   visions
backcall==0.2.0
    # via ipython
beautifulsoup4==4.11.1
    # via nbconvert
binaryornot==0.4.4
    # via cookiecutter
bleach==5.0.1
    # via nbconvert
certifi==2022.9.24
    # via requests
cffi==1.15.1
    # via cryptography
chardet==5.0.0
    # via binaryornot
charset-normalizer==2.1.1
    # via requests
click==8.1.3
    # via
    #   cookiecutter
    #   flytekit
    #   papermill
cloudpickle==2.2.0
    # via flytekit
cookiecutter==2.1.1
    # via flytekit
croniter==1.3.7
    # via flytekit
cryptography==38.0.1
    # via pyopenssl
cycler==0.11.0
    # via matplotlib
dataclasses-json==0.5.7
    # via flytekit
debugpy==1.6.3
    # via ipykernel
decorator==5.1.1
    # via
    #   ipython
    #   retry
defusedxml==0.7.1
    # via nbconvert
deprecated==1.2.13
    # via flytekit
diskcache==5.4.0
    # via flytekit
docker==6.0.0
    # via flytekit
docker-image-py==0.1.12
    # via flytekit
docstring-parser==0.15
    # via flytekit
entrypoints==0.4
    # via
    #   jupyter-client
    #   papermill
executing==1.1.1
    # via stack-data
fastjsonschema==2.16.2
    # via nbformat
flyteidl==1.1.22
    # via flytekit
flytekit==1.2.1
    # via
    #   -r ../../../common/requirements-common.in
    #   flytekitplugins-deck-standard
    #   flytekitplugins-papermill
flytekitplugins-deck-standard==1.2.1
    # via -r ../../../common/requirements-common.in
flytekitplugins-papermill==1.2.1
    # via -r <http://requirements.in|requirements.in>
fonttools==4.37.4
    # via matplotlib
googleapis-common-protos==1.56.4
    # via
    #   flyteidl
    #   grpcio-status
grpcio==1.50.0
    # via
    #   flytekit
    #   grpcio-status
grpcio-status==1.48.0
    # via
    #   -r <http://requirements.in|requirements.in>
    #   flytekit
htmlmin==0.1.12
    # via pandas-profiling
idna==3.4
    # via requests
imagehash==4.3.1
    # via visions
importlib-metadata==5.0.0
    # via
    #   flytekit
    #   keyring
    #   markdown
    #   nbconvert
ipykernel==6.16.0
    # via flytekitplugins-papermill
ipython==8.5.0
    # via ipykernel
jaraco-classes==3.2.3
    # via keyring
jedi==0.18.1
    # via ipython
jinja2==3.1.2
    # via
    #   cookiecutter
    #   jinja2-time
    #   nbconvert
    #   pandas-profiling
jinja2-time==0.2.0
    # via cookiecutter
joblib==1.1.1
    # via
    #   flytekit
    #   pandas-profiling
    #   phik
jsonschema==4.16.0
    # via nbformat
jupyter-client==7.4.2
    # via
    #   ipykernel
    #   nbclient
jupyter-core==4.11.1
    # via
    #   jupyter-client
    #   nbconvert
    #   nbformat
jupyterlab-pygments==0.2.2
    # via nbconvert
keyring==23.9.3
    # via flytekit
kiwisolver==1.4.4
    # via matplotlib
markdown==3.4.1
    # via flytekitplugins-deck-standard
markupsafe==2.1.1
    # via
    #   jinja2
    #   nbconvert
marshmallow==3.18.0
    # via
    #   dataclasses-json
    #   marshmallow-enum
    #   marshmallow-jsonschema
marshmallow-enum==1.5.1
    # via dataclasses-json
marshmallow-jsonschema==0.13.0
    # via flytekit
matplotlib==3.5.3
    # via
    #   -r ../../../common/requirements-common.in
    #   missingno
    #   pandas-profiling
    #   phik
    #   seaborn
matplotlib-inline==0.1.6
    # via
    #   ipykernel
    #   ipython
missingno==0.5.1
    # via pandas-profiling
mistune==2.0.4
    # via nbconvert
more-itertools==9.0.0
    # via jaraco-classes
multimethod==1.8
    # via
    #   pandas-profiling
    #   visions
mypy-extensions==0.4.3
    # via typing-inspect
natsort==8.2.0
    # via flytekit
nbclient==0.7.0
    # via
    #   nbconvert
    #   papermill
nbconvert==7.2.1
    # via flytekitplugins-papermill
nbformat==5.7.0
    # via
    #   nbclient
    #   nbconvert
    #   papermill
nest-asyncio==1.5.6
    # via
    #   ipykernel
    #   jupyter-client
    #   nbclient
networkx==2.8.7
    # via visions
numpy==1.23.4
    # via
    #   imagehash
    #   matplotlib
    #   missingno
    #   pandas
    #   pandas-profiling
    #   patsy
    #   phik
    #   pyarrow
    #   pywavelets
    #   scipy
    #   seaborn
    #   statsmodels
    #   visions
packaging==21.3
    # via
    #   docker
    #   ipykernel
    #   marshmallow
    #   matplotlib
    #   nbconvert
    #   statsmodels
pandas==1.4.4
    # via
    #   flytekit
    #   pandas-profiling
    #   phik
    #   seaborn
    #   statsmodels
    #   visions
pandas-profiling==3.3.0
    # via flytekitplugins-deck-standard
pandocfilters==1.5.0
    # via nbconvert
papermill==2.4.0
    # via flytekitplugins-papermill
parso==0.8.3
    # via jedi
patsy==0.5.3
    # via statsmodels
pexpect==4.8.0
    # via ipython
phik==0.12.2
    # via pandas-profiling
pickleshare==0.7.5
    # via ipython
pillow==9.2.0
    # via
    #   imagehash
    #   matplotlib
    #   visions
plotly==5.10.0
    # via flytekitplugins-deck-standard
prompt-toolkit==3.0.31
    # via ipython
protobuf==3.20.3
    # via
    #   flyteidl
    #   flytekit
    #   googleapis-common-protos
    #   grpcio-status
    #   protoc-gen-swagger
protoc-gen-swagger==0.1.0
    # via flyteidl
psutil==5.9.3
    # via ipykernel
ptyprocess==0.7.0
    # via pexpect
pure-eval==0.2.2
    # via stack-data
py==1.11.0
    # via retry
pyarrow==6.0.1
    # via flytekit
pycparser==2.21
    # via cffi
pydantic==1.9.2
    # via pandas-profiling
pygments==2.13.0
    # via
    #   ipython
    #   nbconvert
pyopenssl==22.1.0
    # via flytekit
pyparsing==3.0.9
    # via
    #   matplotlib
    #   packaging
pyrsistent==0.18.1
    # via jsonschema
python-dateutil==2.8.2
    # via
    #   arrow
    #   croniter
    #   flytekit
    #   jupyter-client
    #   matplotlib
    #   pandas
python-json-logger==2.0.4
    # via flytekit
python-slugify==6.1.2
    # via cookiecutter
pytimeparse==1.1.8
    # via flytekit
pytz==2022.5
    # via
    #   flytekit
    #   pandas
pywavelets==1.4.1
    # via imagehash
pyyaml==6.0
    # via
    #   cookiecutter
    #   flytekit
    #   pandas-profiling
    #   papermill
pyzmq==24.0.1
    # via
    #   ipykernel
    #   jupyter-client
regex==2022.9.13
    # via docker-image-py
requests==2.28.1
    # via
    #   cookiecutter
    #   docker
    #   flytekit
    #   pandas-profiling
    #   papermill
    #   responses
responses==0.22.0
    # via flytekit
retry==0.9.2
    # via flytekit
scipy==1.9.2
    # via
    #   imagehash
    #   missingno
    #   pandas-profiling
    #   phik
    #   seaborn
    #   statsmodels
seaborn==0.11.2
    # via
    #   missingno
    #   pandas-profiling
six==1.16.0
    # via
    #   asttokens
    #   bleach
    #   grpcio
    #   patsy
    #   python-dateutil
sortedcontainers==2.4.0
    # via flytekit
soupsieve==2.3.2.post1
    # via beautifulsoup4
stack-data==0.5.1
    # via ipython
statsd==3.3.0
    # via flytekit
statsmodels==0.13.2
    # via pandas-profiling
tangled-up-in-unicode==0.2.0
    # via
    #   pandas-profiling
    #   visions
tenacity==8.1.0
    # via
    #   papermill
    #   plotly
text-unidecode==1.3
    # via python-slugify
textwrap3==0.9.2
    # via ansiwrap
tinycss2==1.2.1
    # via nbconvert
toml==0.10.2
    # via responses
tornado==6.2
    # via
    #   ipykernel
    #   jupyter-client
tqdm==4.64.1
    # via
    #   pandas-profiling
    #   papermill
traitlets==5.5.0
    # via
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
types-toml==0.10.8
    # via responses
typing-extensions==4.4.0
    # via
    #   flytekit
    #   pydantic
    #   typing-inspect
typing-inspect==0.8.0
    # via dataclasses-json
urllib3==1.26.12
    # via
    #   docker
    #   flytekit
    #   requests
    #   responses
visions[type_image_path]==0.7.5
    # via pandas-profiling
wcwidth==0.2.5
    # via prompt-toolkit
webencodings==0.5.1
    # via
    #   bleach
    #   tinycss2
websocket-client==1.4.1
    # via docker
wheel==0.37.1
    # via
    #   -r ../../../common/requirements-common.in
    #   flytekit
wrapt==1.14.1
    # via
    #   deprecated
    #   flytekit
zipp==3.9.0
    # via importlib-metadata
Also not sure why your requirements has flytekit 0.32.3 but your local system has flytekit 1.2.1
The code works locally but on demo, it’s still throwing the error. @Kevin Su, has this issue not been resolved in 1.2.1 release?
k

Kevin Su

10/19/2022, 11:34 PM
it should work in 1.1.0. there are some breaking changes in 1.2.1. will fix it in 1.2.2
p

Padma Priya M

10/27/2022, 7:21 AM
hi, is this issue resolved in 1.2.2 ? i tried with 1.1.0, facing same issue. not sure if i am doing something wrong. installed all the requirements which u have mentioned by changing only the flytekit version to 1.1.0 and then registered using ghcr.io/flyteorg/flytecookbook:papermilltasks-latest image. the workflow is getting registered successfully but when i create an execution getting this error.
s

Samhita Alla

10/27/2022, 8:43 AM
@Padma Priya M, papermilltasks-latest will have the latest flytekit image. Can you first try running the code locally with flytekit 1.1.0?
p

Padma Priya M

10/27/2022, 9:20 AM
it is working in local with flytekit 1.1.0
s

Samhita Alla

10/27/2022, 9:22 AM
Can you build the image with flytekit 1.1.0 and supply it to the pyflyte run command?
k

Kevin Su

10/27/2022, 9:37 AM
@Padma Priya M sorry, we released early for some reason, so 1.2.2 and 1.2.3 don’t have the fix. I just created a pr, you can give it a try.
pip install git+<https://github.com/flyteorg/flytekit@>"papermill-bug-1#egg=flytekitplugins-papermill&subdirectory=plugins/flytekit-papermill"
View count: 3