https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
ask-the-community
  • r

    Rupsha Chaudhuri

    06/14/2022, 5:27 PM
    Hi team.. Can
    dynamic
    be cached?
    n
    e
    • 3
    • 9
  • e

    Evan Sadler

    06/14/2022, 8:52 PM
    I am trying to use the flyte demo locally (OSX intel chip) using the
    --remote
    command. I am running into an issue. I am not sure where to start looking, so any advice is much appreciated. Inside of a new venv
    $git clone <https://github.com/flyteorg/flytesnacks>
    $cd flytesnacks/cookbook
    $pip install -r core/requirements.txt
    
    $flytectl demo start
    $pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
    And then I get this
    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
    
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNAVAILABLE
    	details = "failed to connect to all addresses"
    	debug_error_string = "{"created":"@1655239314.572674000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3129,"referenced_errors":[{"created":"@1655239314.572673000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
    k
    e
    +5
    • 8
    • 45
  • r

    Rupsha Chaudhuri

    06/15/2022, 8:38 PM
    Hey folks.. quick question: If flyte is deployed on a k8s cluster, can it submit spark jobs on a different k8s cluster?
    šŸ‘€ 1
    k
    k
    • 3
    • 14
  • b

    Brian Dobray

    06/17/2022, 2:35 AM
    Hi all, is there an equivalent for
    list-active-launch-plans
    for
    flytectl
    ? I really really miss this right now:
    flyte-cli \
        -h HOST \
        -d ENV \
        -p PROJECT \
        list-active-launch-plans
    k
    p
    • 3
    • 7
  • r

    Ruksana Kabealo

    06/17/2022, 3:58 PM
    Hi everyone! Has anyone experienced trouble with uploading to Minio? I'm getting an error where my task, which is trying to save a report to minio, can't connect to the minio endpoint. My deployment is a local kubeadm cluster with a single node, multiple pods. Using kubectl get pods -A I can see that the minio pod is up and running. I also can see the minio dashboard at localhost:30088. Any help would be much appreciated!
  • o

    Oliver Nguyen

    06/21/2022, 3:46 AM
    Hi everyone. I am trying to write a task to Insert a row into Postgres table. Because there is very little guide on Insert, so can anyone provide me any documentation on this issue? Thanks you for help.
    k
    k
    • 3
    • 10
  • k

    krishna Yerramsetty

    06/21/2022, 5:31 PM
    Hi all. I have been trying Flyte for the past week or so and I love it! I am new to ML Ops style tools and workflows, so I don't think I get all of the details yet. Question on the sandbox?: Is there a way to download the task outputs to the sandbox container? Right now my outputs for a task that creates a directory is this:
    {1 item
    o0:{2 items
    type:"multi-part blob"
    uri:"<s3://my-s3-bucket/hi/ag5dp94bs4c846cn847s-n0-0/628c0053bbedf3f1db4d44362f48857e>"
    }
    }
    Not sure where this multi-part blob resides
    ā¤ļø 3
    k
    k
    y
    • 4
    • 11
  • o

    Oliver Nguyen

    06/22/2022, 4:13 AM
    I want to deploy spark plugin in the Flyte backend. Step 1: I have stared a Flyte demo cluster by flytectl demo start Step 2: Install spark operator Step 3: Create ā€œvalues-override.yamlā€ file and upgrade the Flyte helm
    helm upgrade flyte flyteorg/flyte-core -f <https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-core/values-sandbox.yaml> -f values-override.yaml -n flyte
    In step 3. I get the following error
    Error: failed to download "flyteorg/flyte-core"
    After checking, the flyte-core not installed after running the ā€œflytectl demo startā€ command, and I not found the flyte-core repository in flyteorg. The flyte-core have still in use or has it been dropped?
    k
    s
    y
    • 4
    • 20
  • s

    seunggs

    06/23/2022, 10:23 PM
    Is there a REST API for the flytectl commands by any chance? (e.g. is there an API endpoint you can hit to run something like
    flytectl create project
    instead of running it as a CLI command?)
    k
    k
    • 3
    • 23
  • k

    Ketan (kumare3)

    06/23/2022, 10:29 PM
    yes
  • k

    Ketan (kumare3)

    06/23/2022, 10:29 PM
    everything has an api
    šŸ‘ 1
  • n

    Nada Saiyed

    06/23/2022, 11:43 PM
    hello.. new flyte user here.. had a question about how can i configure different tasks to use different images in a workflow? i am aware that i can do this by specifying
    @task(
        container_image="{{.image.img1.fqn }}:{{.image.img1.version}}"
    )
    and the value of
    image.img1.version
    can be specified in a config file.. but how is that config linked to tasks while packaging the workflow?
    y
    n
    +2
    • 5
    • 60
  • o

    Oliver Nguyen

    06/27/2022, 3:57 AM
    I want to register my workflow. I have done the following steps Step 1: create project. Example:
    flytectl create project --name myflyteproject --id myflyteproject --description "myflyteproject description" --labels app=myflyte
    Step 2: build docker image Example:
    docker build -t my-flyte-img:v1 .
    docker tag my-flyte-img:v1 <http://ghcr.io/nguyenhuuvan51703223/my-flyte-img:v1|ghcr.io/nguyenhuuvan51703223/my-flyte-img:v1>
    docker push <http://ghcr.io/nguyenhuuvan51703223/my-flyte-img:v1|ghcr.io/nguyenhuuvan51703223/my-flyte-img:v1>
    Step 3: Example:
    pyflyte -c sandbox.config --pkgs core serialize --in-container-config-path /root/sandbox.config --local-source-root /Users/admin/hvan/myproject/flytesnacks/cookbook --image <http://ghcr.io/nguyenhuuvan51703223/my-flyte-img:v1|ghcr.io/nguyenhuuvan51703223/my-flyte-img:v1> workflows -f _pb_output/
    Step 4: Example
    flytectl register files _pb_output/* -p myflyteproject -d development --version v1 k8sServiceAccount demo --outputLocationPrefix <s3://my-s3-bucket/raw_data>
    Step 5: open UI and launch workflow I get the error:
    [1/1] currentAttempt done. Last Error: USER::containers with unready status: [a5k8lnk9s5qh98szvgc8-n0-0]|Back-off pulling image "<http://ghcr.io/nguyenhuuvan51703223/my-flyte-img:v1|ghcr.io/nguyenhuuvan51703223/my-flyte-img:v1>
    Did I miss or take wrong on any step?? Do I need push my image to my docker registry? Do I need define my repository name somewhere?
    k
    • 2
    • 5
  • c

    caioau

    06/27/2022, 1:19 PM
    Hello! I'm having some issues with flyte when istio sidecars are present. When the flyte task container finishes, the pod does not terminate (because the istio sidecar is active) so it is stuck forever. Have you come across this? Do you have any tips or tricks? Thanks very much!
    b
    k
    • 3
    • 11
  • r

    Rupsha Chaudhuri

    06/27/2022, 4:41 PM
    Hi team... quick question. If a task calls a task, do they both not show up in the Flyte console? I only see the outer task
    y
    • 2
    • 12
  • s

    Slackbot

    06/27/2022, 8:28 PM
    This message was deleted.
  • r

    Rupsha Chaudhuri

    06/27/2022, 8:58 PM
    Hey folks.. so I'm exploring how to set up the CI/CD pipeline now that I have some workflows running locally. In order to do this I was experimenting with the flyte-sandbox docker image to register the new version of the workflows and so on. I'm running into an error though
    flytectl get workflows --admin.endpoint <remote_host>:30080 -p <project_name> -d development
    is successful
    docker run --rm --entrypoint flytectl <http://cr.flyte.org/flyteorg/flyte-sandbox|cr.flyte.org/flyteorg/flyte-sandbox> get workflows --admin.endpoint <remote_host>:30080 -p <project_name> -d development
    throws an error
    time="2022-06-27T20:18:25Z" level=info msg="[0] Couldn't find a config file []. Relying on env vars and pflags."
    {"json":{},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"","ts":"2022-06-27T20:18:26Z"}
    {"json":{},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-06-27T20:18:26Z"}
    {"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-06-27T20:18:26Z"}
    Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
    {"json":{},"level":"error","msg":"rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"","ts":"2022-06-27T20:18:27Z"}
    y
    k
    • 3
    • 9
  • n

    Nada Saiyed

    06/27/2022, 9:11 PM
    hello! i am trying to run a workflow which has a task/step that fetches data from sandbox’s Minio and splits it into train/test. But the pod keeps dying with this error message
    Pod failed. No message received from kubernetes.
    [atg6lnllwptbr4j6thwc-n0-0] terminated with exit code (137). Reason [Error]. Message: 
    
    .
    the flyte cluster is local sandbox cluster
    k
    s
    y
    • 4
    • 27
  • s

    seunggs

    06/27/2022, 9:18 PM
    Hi, where can I check the latest Flyte helm chart version?
    k
    y
    • 3
    • 3
  • r

    Rupsha Chaudhuri

    06/28/2022, 6:46 PM
    I'm still running into the issue here even with flytekit 1.0.3. Create project is asking for project?
    y
    • 2
    • 8
  • r

    Rupsha Chaudhuri

    06/28/2022, 11:53 PM
    Hi team.. I'm trying to serialize the workflows in order to run
    flytectl register files
    .. However pyflyte is not serializing anything. This is what I see when I run
    pyflyte serialize...
    in the container. Any ideas what's going on?
    pyflyte serialize workflows -f /tmp/_pb_output
    {"asctime": "2022-06-28 23:51:47,894", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
    {"asctime": "2022-06-28 23:51:48,124", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
    {"asctime": "2022-06-28 23:51:48,125", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
    {"asctime": "2022-06-28 23:51:48,125", "name": "flytekit", "levelname": "INFO", "message": "Setting protocol to file"}
    {"asctime": "2022-06-28 23:51:48,125", "name": "flytekit", "levelname": "INFO", "message": "We won't register bigquery handler for structured dataset because we can't find the packages google-cloud-bigquery-storage and google-cloud-bigquery"}
    Serializing Flyte elements with image 
    Writing output to /tmp/_pb_output
    Loading packages [] under source root /root
    Successfully serialized 0 flyte objects
    k
    • 2
    • 3
  • r

    Rupsha Chaudhuri

    06/29/2022, 5:53 PM
    Hey team.. so we have our custom UI that uses the REST API to trigger workflows in the Flyte backend. Now every time we bump the version of the workflows, the REST APIs invoked by the UI need to change as well to use the latest version. Is there a "latest" tag or something like that that can be used to alleviate this problem?
    k
    a
    • 3
    • 31
  • t

    Tan Li

    06/30/2022, 7:17 AM
    Hi community, I was trying to setup flyte-sandbox locally on my k3s cluster. I followed this instruction https://docs.flyte.org/en/latest/deployment/sandbox.html#deployment . First I was stuck at the point that
    svclb-flyte-deps-contour-envoy-c519e84f-lnpvm
    keeps pending in the namespace kube-system. I worked around this by upgradeing the flyte-deps config. However, I am still confronting the envoy pod pending issue. Terminal output are in the reply thread. Anyone know how should I fix this? In general, it seems to be some port allocation issue, but I really don’t have any idea how to debug this futher, lol. Really appreciate it! šŸ™
    šŸ‘ 2
    k
    k
    • 3
    • 15
  • a

    Aleksei Potov

    06/30/2022, 9:42 PM
    Is there a way to setup "auto-cleanup" of old executions (e.g. older than couple days)? Asking because pods are not deleted until execution is deleted, so we are ending up with a like a gazillion of completed pods. Or there's another solution for not having completed pods lying around?
    k
    • 2
    • 10
  • f

    Fredrick

    07/01/2022, 6:25 PM
    Hi flyte team, I integrated our OIDC provider for flyteadmin auth. But its failing during OIDC callback with the following error. It looks to be cookie that it generates using the auth_code becomes bigger than 4096 [gorilla securecookie implementation](https://github.com/gorilla/securecookie/blob/master/securecookie.go#L256-L259). Any ideas on how to fix this ?
    {
      "json": {
        "src": "handlers.go:161"
      },
      "level": "error",
      "msg": "Error setting encrypted JWT cookie [SECURE_COOKIE_ERROR] Error creating secure cookie, caused by: securecookie: the value is too long",
      "ts": "2022-07-01T07:41:05Z"
    }
    k
    k
    • 3
    • 2
  • a

    Andrew Achkar

    07/06/2022, 5:18 PM
    With the way packaging and registration works with flyte, what is the recommended way to provide a common set of library methods across multiple flyte projects? (eg. our team wants to define a CustomTask that multiple projects want to use). Is there any example of this we can reference?
    k
    k
    • 3
    • 5
  • s

    seunggs

    07/08/2022, 2:11 AM
    Hi using Google for flyte console auth and I can’t find documentation on how to use the refresh token - it logs me out constantly otherwise?
    y
    k
    p
    • 4
    • 14
  • s

    seunggs

    07/08/2022, 2:11 AM
    If anyone could point to the doc on this, it would be much appreciated!
  • s

    Stefan Avesand

    07/13/2022, 1:57 PM
    Has anyone tried running Dask together with Flyte?
    k
    b
    r
    • 4
    • 10
  • s

    Slackbot

    07/15/2022, 5:23 PM
    This message was deleted.
    k
    a
    +3
    • 6
    • 20
Powered by Linen
Title
s

Slackbot

07/15/2022, 5:23 PM
This message was deleted.
k

Ketan (kumare3)

07/15/2022, 5:38 PM
i think you do not have the right endpoint configured
please configure the right endpoint in the ~/.flyte/config.yaml
šŸ‘ 1
a

austin

07/15/2022, 7:15 PM
i also get that error:
<_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"
etc…. even when the script has [at least initially] connected to the right endpoint. Not sure exactly the source of flakiness. For @Taeef Najib, I agree ensuring config is correct is the good place to start.
šŸ‘ 1
n

Niels Bantilan

07/15/2022, 8:08 PM
can you
cat ~/.flyte/config.yaml
and copy-paste it here?
šŸ‘ 1
a

austin

07/15/2022, 8:14 PM
To hijack Taeef’s thread … https://github.com/flyteorg/flyte/actions/runs/2678784708 … see the artifact ā€˜config.yaml’ … this one failed ( after a short bit ). When I ran fewer tests: https://github.com/flyteorg/flyte/runs/7363730667 ( it was successful ).
and, by tests, I mean the contents of: https://github.com/flyteorg/flyte/blob/opta-aws/boilerplate/flyte/end2end/run-tests.py
my current thinking is that it is also possible to get that error from resource constraints-or-maybe-misbehaving-workflows/executions [ haven’t isolated specific issue ], but doesn’t seem to be the
config.yaml
y

Yee

07/15/2022, 9:21 PM
yeah the error is a generic connection error, can happen for a number of reasons
the most common reason i feel like we’ve seen is the wrong address, or not adding
dns:///
in front of the domain, or using/not using insecure
a

austin

07/15/2022, 9:31 PM
sure … but wonder what cases we could put messaging in place to debug that — ex: when first parsing the config file ex: ā€œERROR: the proper format of
endpoint
field is ’dns://<DNS_OF_SERVICE> … ā€˜dns://’ is missingā€ā€¦. [ but I’m not sure if you could address with just IP, and in that case, dns:// might not be required ]. Or other things like that….
I think tests will start in a couple minutes…. Will see whether they fail [ I believe VERY likely to fail ]. https://github.com/flyteorg/flyte/runs/7365263197?check_suite_focus=true
t

Taeef Najib

07/17/2022, 1:36 AM
Thank you all for all the suggestions. However, I might have made some mistake in installing flyte and getting it to work on my mac. I tried a Windows laptop to make sure that it’s not the hardware problem. On windows, I get the same error. I’d like to mention that I didn’t install flytectl on my machines. I tried to install it on windows with the following command: First
bash
and then
curl -sL <https://ctl.flyte.org/install>
but no luck. I see a file called
flytectl
in the bin folder. But that is not editable or readable on vs code. I am very inexperienced into this, but how do I find
config.yaml
for flyte on my windows machine? or how do I correctly install it using cmd (on vs code)?
a

austin

07/17/2022, 4:21 AM
@Taeef Najib flytectl likely would be a binary, so I wouldn’t expect it to be editable/readable with VSCode.
šŸ‘ 1
unfortunately, I can’t help you with windows šŸ˜•, but maybe someone else can
šŸ‘ 1
k

Ketan (kumare3)

07/17/2022, 5:59 AM
It is a golang binary and should work on windows, but I don't have a windows machine
šŸ‘ 1
t

Taeef Najib

07/17/2022, 12:09 PM
Under the bin folder, there’s just one binary file called flytectl. Am I not supposed to have a folder where I can find config.yaml? @austin and @Ketan (kumare3)
k

Ketan (kumare3)

07/17/2022, 2:57 PM
You have to do flytectl config init - check the docs https://docs.flyte.org/projects/flytectl/en/latest/index.html
šŸ‘ 1
t

Taeef Najib

07/17/2022, 6:47 PM
Thanks @Ketan (kumare3) šŸ™
a

austin

07/17/2022, 7:25 PM
@Taeef Najib — those docs solved things for you? Anything else that could be better highlighted/documented?
šŸ‘ 1
t

Taeef Najib

07/17/2022, 9:00 PM
@austin even after installing flytectl using my vs code terminal (cmd) I can’t upgrade or check the version of flytectl. Under the bin folder, I can see the flytectl file, but terminal returns this error:
'flytectl' is not recognized as an internal or external command
That’s where I’m stuck. Do you have any suggestions? šŸ˜ž When I use
curl -sL <https://ctl.flyte.org/install> | bash
I get this:
flyteorg/flytectl info checking GitHub for latest tag
flyteorg/flytectl info found version: 0.6.4 for v0.6.4/Linux/x86_64
flyteorg/flytectl info installed ./bin/flytectl
View count: 7