• m

    Marcus Isnard

    2 months ago
    I have deployed the Flyte Sandbox to a remote machine -- UI is accessible and everything is up. I have set the endpoint in ~/.flyte/config.yaml. When I run a workflow using PyFlyte and a custom container in a registry local to my machine, I get the following error:
    flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmpby0anszu/script_mode.tar.gz to <http://localhost:30084/my-s3-bucket/ff/flytesnacks/development/MOAHDD5B6MWAWQCHZQVZWQX6UU%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220715%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220715T180620Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=d807a4f73adf126a579aa7dcbe1800e8654559f37c8728500b61d20b067d804e> (recursive=False).
    
    Original exception: HTTPConnectionPool(host='localhost', port=30084): Max retries exceeded with url: /my-s3-bucket/ff/flytesnacks/development/MOAHDD5B6MWAWQCHZQVZWQX6UU%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220715%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220715T180620Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=d807a4f73adf126a579aa7dcbe1800e8654559f37c8728500b61d20b067d804e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f440c5cd910>: Failed to establish a new connection: [Errno 111] Connection refused'))
    m
    Niels Bantilan
    +2
    16 replies
    Copy to Clipboard
  • Taeef Najib

    Taeef Najib

    2 months ago
    I’m posting this separately because I think the previous message would confuse others. I use windows. I tried installing flytectl using my vs code terminal (cmd) but I can’t upgrade or check the version of flytectl. I installed it inside my project folder, then I also installed it inside my user folder (C:\Users\user) and also added the PATH (C:\Users\user\bin) to my environment. Under the bin folder, I can see the flytectl file, but when I try the code
    flytectl upgrade
    or
    flytectl version
    my terminal returns this error:
    'flytectl' is not recognized as an internal or external command
    That’s where I’m stuck. Do you have any suggestions? 😞 BTW, I used:
    curl -sL <https://ctl.flyte.org/install> | bash
    and got this:
    flyteorg/flytectl info checking GitHub for latest tag
    flyteorg/flytectl info found version: 0.6.4 for v0.6.4/Linux/x86_64
    flyteorg/flytectl info installed ./bin/flytectl
    My concern is whether flytectl can be used on Windows or not! Because someone else tried to reproduce this error using a Windows machine and received the same error. Does any Windows user could use flytectl?
    Taeef Najib
    Samhita Alla
    +1
    7 replies
    Copy to Clipboard
  • k

    karthikraj

    2 months ago
    Hi, I have deloyed the helm-charts in AWS by following the charts available here. When I deploy, the target groups for HTTP and GRPC are being created in AWS but both backend-protocol version are HTTP, so the grpc target group is never coming healthy. Any solution here? Do I need to edit anything here? Do i need to add a separate ingress rules for HTTP and GRPC?
    k
    Ketan (kumare3)
    +1
    9 replies
    Copy to Clipboard
  • Taeef Najib

    Taeef Najib

    2 months ago
    I was trying to test Pima Diabetes project using Flyte as per the documentation. When I try:
    pyflyte run --remote model.py:diabetes_xgboost_model
    The first task (split_traintest_dataset) fails and I get this error:
    [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
    [ffeb1a0e5b4f44d81a8c-n0-0] terminated with exit code (1). Reason [Error]. Message: 
    cked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/root/model.py", line 14, in <module>
        import joblib
    ModuleNotFoundError: No module named 'joblib'
    Traceback (most recent call last):
      File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
        sys.exit(fast_execute_task_cmd())
      File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 506, in fast_execute_task_cmd
        subprocess.run(cmd, check=True)
      File "/usr/local/lib/python3.10/subprocess.py", line 524, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ffeb1a0e5b4f44d81a8c/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ffeb1a0e5b4f44d81a8c/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/kh/ffeb1a0e5b4f44d81a8c-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/kh/ffeb1a0e5b4f44d81a8c-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/bc/flytesnacks/development/NBHGKLLB7JMWQJQQLZNV66UGGU======/scriptmode.tar.gz>', '--dynamic-dest-dir', '.', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'model', 'task-name', 'split_traintest_dataset']' returned non-zero exit status 1.
    .
    `Any idea what's wrong? I faced a similar issue while trying the MNIST digits classification project.
    Taeef Najib
    Ketan (kumare3)
    +1
    5 replies
    Copy to Clipboard
  • Rupsha Chaudhuri

    Rupsha Chaudhuri

    2 months ago
    Hi team.. I'm trying to schedule a launchplan based on the example here. However when I execute the activation command
    flytectl update launchplan -p flyteexamples -d development {{ name_of_lp }} --activate
    it's erroring out
    launch plan version wasn't passed
    Rupsha Chaudhuri
    y
    2 replies
    Copy to Clipboard
  • Calvin Leather

    Calvin Leather

    1 month ago
    Hey we ran into an interesting problem today... we have a map task that calls a C binary that uses a large reference dataset from disk to do some computations on a new smaller dataset. It keeps failing with a rather mysterious error:
    [0]: code:"ResourceDeletedExternally" message:"resource not found, name [e2e-workflows-development/fb2xnzxy-n2-0-0]. reason: pods \"fb2xnzxy-n2-0-0\" not found"
    We then checked control plan logs and they suggested the pod was being evicted due to memory pressure (137 = k8s OOM status code):
    "containerStatuses": [
                    {
                        "name": "fb2xnzxy-n2-0-0",
                        "state": {
                            "terminated": {
                                "exitCode": 137,
    ....
    However when we look at grafana, we see that memory used is really low, way below requests/limits... however, we found that the memory cache was quite high. We then found a k8s issue about memory cache being incorrectly counted as "used" memory by kubelet when it looks at memory pressure. Note quite a flyte issue, more of a k8s issue, but the log was a bit mysterious and we're still figuring out resolution.
    Calvin Leather
    Louis DiNatale
    +4
    49 replies
    Copy to Clipboard
  • Taeef Najib

    Taeef Najib

    1 month ago
    I'm on Windows 10. I've installed
    flytekit
    using:
    pip install flytekit
    tried to reinstall it, used conda to install it, removed it and then installed it again using pip. But when I use
    flytekit status
    on my vscode terminal I get this error:
    'flytekit' is not recognized as an internal or external command,
    operable program or batch file.
    I've added these PATHS to my environment variables [the image is attached] Can anyone please give me a clue? Why is
    flytekit
    or any of its features like
    pyflyte
    is not recognized on my terminal (either vscode or cmd)? Any help would be really appreciated. Thanks.
    Taeef Najib
    Ketan (kumare3)
    +2
    9 replies
    Copy to Clipboard
  • Ekku Jokinen

    Ekku Jokinen

    1 month ago
    Hello! I’m very new to Flyte, and I’m trying to deploy some workflows to AWS to make processing faster. I have a deployment running, console is accessible and all. Now I need to deploy the custom images to AWS, and I’m wondering what would be the recommended way to do so. I understand the images should be pushed to a registry, should I go for AWS ECR, perhaps Docker hub, or something else? And after pushing the images to a registry, how do I configure the flyte cluster to be able to access those images? Best, Ekku, CTO & Co-Founder @ inven
    Ekku Jokinen
    Ketan (kumare3)
    +1
    6 replies
    Copy to Clipboard
  • Eduardo Apolinario (eapolinario)

    Eduardo Apolinario (eapolinario)

    1 month ago
    @Calvin Leather / @Louis DiNatale, just following up on the resource overriding issue we talked about earlier today. If I remember correctly, you were seeing a discrepancy between setting task resource limits using the @task decorator vs setting default values via
    flytectl update task-resource-attribute
    and the task resource defaults defined in the helm chart values file. Just so we can dive deeper on this, can you tell us a bit about what you tried? We're probably missing some documentation around this area.
    Eduardo Apolinario (eapolinario)
    Louis DiNatale
    +2
    16 replies
    Copy to Clipboard
  • Ailin Yu

    Ailin Yu

    1 month ago
    Hello! Might I be given permissions to push branches to flyteplugins so I may contribute to this change? Not sure how this is usually done around here 🙏
    ERROR: Permission to flyteorg/flyteplugins.git denied to niliayu.
    fatal: Could not read from remote repository.
    
    Please make sure you have the correct access rights
    and the repository exists.
    Ailin Yu
    k
    +1
    4 replies
    Copy to Clipboard