https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
ask-the-community
  • y

    Yash Panchwatkar

    10/16/2022, 2:37 PM
    hello community.. does flyte have the AWS SQS dependency?? @anyone
    k
    • 2
    • 11
  • k

    karthikraj

    10/16/2022, 8:37 PM
    Hi, I am trying to use kuberay operator. I have created kuberay-operator and ray-cluster using helm chart and it is running in a new namespace. When I am trying to run an example code mentioned in the blog here , getting this below error.
    File "/home/kkanagar/kkr/kuberay_test.py", line 15, in <module>
        def ray_task() -> list[int]:
    TypeError: 'type' object is not subscriptable
    Changing this list[int] to any is working fine in local but when trying to register to remote flyte. Need your help, please
    k
    • 2
    • 3
  • t

    Taeef Najib

    10/17/2022, 12:56 AM
    I was told that we can use regular Python type hinting for files. But in the example: https://docs.flyte.org/projects/cookbook/en/latest/auto/case_studies/ml_training/mnist_classifier/pytorch_single_node_and_gpu.html I see
    from flytekit.types.file import PythonPickledFile
    and in the example: https://docs.flyte.org/projects/cookbook/en/latest/auto/case_studies/ml_training/house_price_prediction/house_price_predictor.html I see
    from flytekit.types.file import JoblibSerializedFile
    These are used to output the model state. Couldn’t we use regular Python type hints for these? May I know why
    PythonPickedFile
    and
    JoblibSerializedFile
    were used in the examples?
    k
    s
    • 3
    • 4
  • n

    Nada Saiyed

    10/17/2022, 3:23 PM
    Hello, I am writing a custom flytekit plugin and I was wondering how can I retrieve the rendered image uri in
    execute()
    of a custom plugin, if i pass a templated value to
    container_image
    . E.g. is my task definition
    @task(task_config=my_task_config, container_image="{{.image.img1.fqn }}:{{.image.img1.version}}"
    I can see that the
    container_image
    is set as a property and if I retrieve it, I see it’s value is still a templated string. How can I get the actual rendered image uri for that task? (i.e. basically I want what this function returns)
    t
    y
    k
    • 4
    • 21
  • l

    Laura Lin

    10/17/2022, 6:04 PM
    Hi, using flyte again after a long break. What is the current equivalent of
    from flytekit.sdk.types import Types
    the
    Types.Generic
    ? This used to be the type for arbitrary dict/json objects but it doesn't seem to be here anymore?
    y
    • 2
    • 5
  • l

    Laura Lin

    10/17/2022, 8:55 PM
    can you use docker images when you run
    pyflyte run
    without the remote option. e.g. if I define a
    @task(container_image="XXXX"
    , can I use this image locally when I run?
    y
    • 2
    • 6
  • k

    karthikraj

    10/17/2022, 10:30 PM
    Hi , I am trying to integrate okta with Flyte. Following this page for setting up appAuth and userAuth. First tried only setting up userAuth in configmap. It worked, okta is enabled. Then included appAuth to configmap and re-deployed, facing the below error in flytescheduler-check container of flytescheduler. Using the same client_id for userAuth and appAuth.
    panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 400 Bad Request
    Response: {"errorCode":"invalid_client","errorSummary":"Invalid value for 'client_id' parameter.","errorLink":"invalid_client","errorId":"oae4BTFncguRFCNZUfFEpIFhA","errorCauses":[]}
    
    goroutine 1 [running]:
    main.main()
            /go/src/github.com/flyteorg/flyteadmin/cmd/scheduler/main.go:12 +0x85
    This is my sample configmap for this.
    configmap:
        auth:
          appAuth:
            authServerType: External
            externalAuthServer:
              baseUrl:  <https://www.oktadev.com/oauth2/default>
            thirdPartyConfig:
              flyteClient:
                clientId:  "a1b2c3xxxxyz9N3" 
                redirectUri: <https://flytedev.myapp.com/callback>
                scopes:
                - offline
                - all
          userAuth:
            openId:
              baseUrl: <https://www.oktadev.com/oauth2/default>
              scopes:
                - profile
                - openid
              clientId: "a1b2c3xxxxyz9N3"
          authorizedUris:
            - <https://flytedev.myapp.com>
            - <http://flyteadmin:80>
            - <http://flyteadmin.flyte.svc.cluster.local:80>
    t
    s
    +2
    • 5
    • 46
  • c

    cryptic

    10/18/2022, 3:46 AM
    this is probably a silly question but i'm looking to serialize
    tf.tensor
    . The
    tf.io.serialize_tensor
    returns a string, should we store it in
    tf.train.Feature
    ? and further put this into
    tf.train.Example
    and use
    SerializeToString()
    method to serialize the proto? Is there a need to store it on to the disk? i'm not sure if storing on disk is possible or not
    k
    s
    r
    • 4
    • 11
  • k

    Ketan (kumare3)

    10/18/2022, 4:04 PM
    join the community sync here - https://zoom.us/j/93875115830?pwd=YWZWOHl1ODRRVjhjVkxSV0pmZkJaZz09
  • c

    Chris Hairfield

    10/18/2022, 7:21 PM
    Hello, I’m using
    fast register
    to develop in the
    sandbox
    environment, and I’d like to configure it to use a storage backend other than
    awscli
    . Is there any documentation for doing this? We run exclusively in GCP and install
    gsutil
    in each of our Flyte-ready Docker images. I’m hoping to avoid installing
    awscli
    in our images just to support running in
    sandbox
    . Thanks!
    k
    e
    • 3
    • 9
  • m

    Martin Hwasser

    10/19/2022, 7:57 AM
    Hi! I have LaunchPlan created in Python that I want to trigger in my CI/CD flow based on file changes. The documentation shows how to create an execution given a spec file, but how do I actually execute a launch plan that’s already created programmatically?
    k
    y
    +2
    • 5
    • 22
  • f

    Fabio Grätz

    10/19/2022, 8:57 AM
    Hey everyone 🙂 I'm currently deploying a flyte sandbox at my new company and am struggling with getting auth to work. I followed this guide. Flyte is deployed, I am redirected to the SSO login screen, after that I can view the flyte console 👍 I can also run
    flytectl get projects
    , am redirected to the browser, and then retrieve the projects.
    pyflyte register ...
    works as well. So far so good. When I manually launch a registered workflow from the console, it doesn't start however.
    k -n development describe <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com> ah9ghn272q7v22dbsvvd
    gives:
    Status:
      Failed Attempts:  9
      Message:          Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"]
      Phase:            0
    Tasks:
      resource_type:TASK project:"3dod" domain:"development" name:"project.workflows.workflow.deploy_model" version:"2"
    Details about config in 🧵 A pointer in the right direction would be much appreciated.
    s
    k
    +5
    • 8
    • 95
  • t

    Tom Melendez

    10/19/2022, 1:42 PM
    Hey Folks, new here but our team loves Flyte! I'm seeing a hang from a fork when I use flyte remote - should this being forking? I'm only just getting into this but if anyone has any quick thoughts or ideas, PLMK. I'll post more info soon. Running python3 in a venv, flytekit 1.1.3, Ubuntu 20.04 LTS on GCP. python3 -v tells me I'm hanging on popen_fork.
    k
    • 2
    • 4
  • f

    Felix Ruess

    10/19/2022, 7:21 PM
    Hi, how can I run a ContainerTask on a GPU enabled node and set
    runtimeClassName
    to
    nvidia
    so that it can actually use the GPU? I added
    requests=Resources(gpu="1")
    but can I add the runtimeClassName?
    k
    m
    • 3
    • 11
  • v

    varsha Parthasarathy

    10/19/2022, 9:46 PM
    Hi team, does anyone know what can be a good default value for type
    FlyteFile
    y
    • 2
    • 8
  • l

    Laura Lin

    10/19/2022, 10:10 PM
    How do you scale up nodes inside a EKS cluster. If I set my desired + min to 1 and then I spawn a bunch of jobs, shouldn't I expect it to create new nodes if I don't hit the max node? But the default auto scaling group doesn't seem to be doing that. Is there something in the helm chart that I should change for that?
    k
    • 2
    • 4
  • l

    Leiqing

    10/20/2022, 2:25 PM
    Hi team, I need to set some env variables (e.g. PATH/PYTHONPATH/LD_LIBRARY_PATH) in an image, but I cannot do so when building the docker image since their values are dynamically determined when building the image, and docker does not allow that. How can I source those env variables on a worker pod before
    pyflyte-fast-execute
    gets invoked? Or is there any other recommended way of doing this?
    l
    k
    • 3
    • 3
  • l

    Luísa Braga Simões

    10/20/2022, 2:47 PM
    Hi all, I want to try the new ray plugin in flyte. Can I do that using a sandbox environment?
    e
    s
    k
    • 4
    • 5
  • a

    Andrew Korzhuev

    10/20/2022, 3:11 PM
    I'm getting this error on flytescheduler running on AWS EKS, which I guess is admin ip:
    Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp ...:81: i/o timeout"
    panic: authentication error! Original Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken, Auth Error: failed to initialized token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp ...:81: i/o timeout"
    Could this be related to auth config? And if so where do I look to fix it?
    k
    p
    • 3
    • 11
  • f

    Frank Shen

    10/20/2022, 10:16 PM
    Hello, I have some common libraries (including pyspark) from both public and private repos that I would like all my flyte workflows to share. To do that, I would like to build a custom flyte docker image. However, I am not able to find a clear documentation on how to do it and how to refer to the custom image when running or registering my flows. Could you please help? Thanks a lot.
    e
    k
    s
    • 4
    • 10
  • k

    karthikraj

    10/21/2022, 2:59 AM
    Hi, I am trying to use kuberay operator and ray cluster with in the Flyte by following this documentation. https://blog.flyte.org/ray-and-flyte#heading-launch-a-ray-cluster I have deployed the kuberay-operator and ray-cluster by following the helm chart docs given here https://github.com/ray-project/kuberay/tree/master/helm-chart. The pods are up and running. I have also enabled the ray plugin in the flyte. When running the pyflyte remotely, I am getting this below error.(I tried deploying the ray pods both in separate namespace as well as in the namespace where flyte pods are running). Need help here on this error.
    raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
            status = StatusCode.INTERNAL
            details = "failed to create workflow in propeller json: error calling MarshalJSON for type *v1alpha1.Inputs: Marshal called with nil"
            debug_error_string = "{"created":"@1666318439.715331527","description":"Error received from peer ipv4:35.172.9.181:443","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"failed to create workflow in propeller json: error calling MarshalJSON for type *v1alpha1.Inputs: Marshal called with nil","grpc_status":13}"
    d
    s
    • 3
    • 12
  • s

    Sanjay Chouhan

    10/21/2022, 9:51 AM
    Hi, I am getting,
    Requested MEMORY default [6Gi] is greater than the limit [150Mi]. Please fix your configuration
    where and how to configure the limit? I am using the AWS EKS setup.
    s
    • 2
    • 6
  • m

    Martin Hwasser

    10/21/2022, 12:29 PM
    Hi, I have a strange error where I am creating several LaunchPlans with different names using
    flytekit.LaunchPlan.get_or_create
    . However, when I try to register this archive with the launchplans, it fails on a particular one with the name:
    a-cst-old
    , giving the error:
    Error: rpc error: code = InvalidArgument desc = missing name
    It’s just this name and the fixed input (string) that’s changed, nothing else, if I prefix this name with something else like, it works. And on a related note, is there a way to delete, not just archive, all launchplans?
    s
    p
    h
    • 4
    • 11
  • l

    Louis DiNatale

    10/21/2022, 3:29 PM
    Hey Flyte team! Can flytekit conditionals be used inside dynamics? I noticed that they say that have to be used within a workflow just trying to understand if its used in a dynamic that is then used in a workflow thats ok.
    s
    • 2
    • 1
  • l

    Laura Lin

    10/21/2022, 3:29 PM
    On the flyte UI, there's an option for
    Only my executions
    filtering. Right now when I click on that, I get
    Request failed with status code 400 invalid value for filters
    What needs to be done for this to work? Do I have to set up OAuth login for the UI?
    s
    s
    p
    • 4
    • 7
  • h

    Hitarth Srivastava

    10/21/2022, 4:38 PM
    With regards to https://github.com/flyteorg/flyte/issues/2992, should I update the link or just remove it? If I have to update it then what procedure should I follow?
    s
    • 2
    • 1
  • f

    Frank Shen

    10/21/2022, 5:24 PM
    Hello, I am having trouble building a custom image that will execute pyspark tasks.
    Last Error: USER::Spark Job Failed with Error: driver container failed with ExitCode: 127, Reason: ContainerCannotRun
    I have the following in the requirements.txt file for spark related dependencies.
    flytekit>=0.20.0
    flytekitplugins-snowflake
    flytekitplugins-spark
    scikit-learn
    xgboost
    unionml
    SHAP
    pandas
    apache-beam
    joblib
    pyyaml==6.0
    boto3==1.9.207
    aws-secretsmanager-caching==1.1.1.5
    aws-requests-auth==0.4.3
    e
    • 2
    • 4
  • f

    Frank Shen

    10/21/2022, 5:25 PM
    Could you tell me what are missing?
  • f

    Frank Shen

    10/21/2022, 5:57 PM
    Hello, Regarding I found this https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html#spark-docker-image, and its example Dockerfile.
    RUN python3 -m venv ${VENV}
    ENV PATH="${VENV}/bin:$PATH"
    
    RUN pip3 install wheel
    
    # Install Python dependencies
    COPY k8s_spark/requirements.txt /root
    RUN pip install -r /root/requirements.txt
    
    RUN flytekit_install_spark3.sh
    k
    • 2
    • 3
  • f

    Frank Shen

    10/21/2022, 5:57 PM
    Could you point me to the full repo location?
Powered by Linen
Title
f

Frank Shen

10/21/2022, 5:57 PM
Could you point me to the full repo location?
View count: 2